canary: detect provider API schema drift; release v0.7.0

The export reads ChatGPT's and Claude's undocumented internal web APIs,
which can change shape without notice; the worst failure for a backup tool
is a silent one (skipped/mis-parsed content with no error). Add a `canary`
command + BaseProvider.check_drift() (overridden by ChatGPT/Claude) that
fetches one listing page + one conversation per provider and asserts only
the normalizer's load-bearing fields — not the full response shape, which
churns harmlessly. The top silent risk it guards is a renamed retrieval-tool
author bypassing the hidden-content collapse. ERROR findings exit non-zero;
WARN findings are surfaced but non-fatal so a backup run is never blocked.

Also retires the in-app watch mode and headless StartOS direction from the
roadmap (tool stays a local, manually-run CLI) and updates docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

CHANGELOG.md

@@ -3,6 +3,18 @@
 All notable changes to this project will be documented here.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [0.7.0] - 2026-06-28
+
+### Added
+- `canary` command — probes the ChatGPT/Claude web APIs for schema drift against the fields the parser actually depends on (one listing page + one conversation per provider), not the full response shape. Flags the silent-failure risks for a backup tool: a renamed retrieval-tool author bypassing the hidden-content collapse, a new `content_type`, drifted message fields, or non-empty Claude `attachments`/`files` the normalizer ignores. ERROR findings exit non-zero; WARN findings are surfaced but non-fatal. See `FUTURE.md` §10.
+- `doctor` now reports a real ChatGPT token-health check ("ChatGPT token active") based on the `/api/auth/session` `error` field. ChatGPT session tokens are JWEs whose `exp` is encrypted and unreadable client-side; the previous decode path could never yield an expiry. The honest signal is `error == "RefreshAccessTokenError"` (verified live), which means the session token is dead even though `expires`/`accessToken` still echo stale values. See `FUTURE.md` §9.
+
+### Fixed
+- `ChatGPTProvider._fetch_access_token` now checks the `/api/auth/session` `error` field and fails fast with a clear "refresh your token" message. Previously it returned the stale `accessToken` present on a dead session, producing a confusing downstream 401 instead of an actionable auth error.
+
+### Removed
+- `auth --from-browser` and the `browser-cookie3` dependency (`src/browser_tokens.py`). Browser cookie auto-extraction is not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave) keys cookies off a SYSTEM-level layer that can't be decrypted off disk without Administrator rights and AV-flagged SYSTEM impersonation, and fails on Brave specifically (symptom: `Unable to get key for cookie decryption`). Auth is manual (DevTools wizard) only. See `FUTURE.md` §2 for the full rationale.
+
 ## [0.6.0] - 2026-06-12
 
 First tagged release. The project was developed through several internal

FUTURE.md

@@ -17,14 +17,18 @@ of these additions straightforward.
 
 Priorities reflect the tool's primary purpose — a trustworthy backup so that
 conversation data is not lost if a provider account is ever closed — plus the
-day-to-day friction of weekly ChatGPT token refresh, and the long-term goal
-of running headless as a StartOS service.
+day-to-day friction of the weekly ChatGPT token refresh. The tool stays a
+local, manually-run CLI; the headless/StartOS direction was dropped
+2026-06-28 (see #7 and #8), which also retires the token-freshness problem
+(manual refresh is sufficient).
 
 **Now (in order):**
 1. ~~Collapse tool retrieval dumps & hidden context~~ — **shipped in v0.6.0**
-   (full-archive re-export still pending; see entry below)
-2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0** (note: runs on
-   whichever machine hosts the browser — Brave is not on this Linux box)
+   (full-archive `export --force` re-export completed 2026-06-13)
+2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0, removed afterward.**
+   Not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave)
+   needs admin + SYSTEM impersonation that AV flags as credential theft, and
+   fails on Brave specifically. Auth is manual (DevTools) — see entry below.
 3. ~~Claude Code session provider~~ — **shipped in v0.6.0**
 4. ~~Archive hygiene — `prune` + `doctor` integrity check~~ — **shipped in v0.6.0**
 
@@ -32,12 +36,23 @@ of running headless as a StartOS service.
 
 5. ~~Binary content downloads~~ — **shipped in v0.6.0**
 6. ~~Per-session download limiter + polite pacing~~ — **shipped in v0.6.0**
-7. Scheduled / watch mode → StartOS service packaging (long-term destination)
+7. ~~Scheduled / watch mode~~ — **dropped 2026-06-28**; the tool stays a
+   manually-run CLI, so no in-app polling loop is needed
+8. ~~StartOS service packaging~~ — **dropped 2026-06-28**; the local CLI is
+   sufficient (source convos live in the cloud and can be re-downloaded;
+   Joplin already syncs encrypted to an offsite S3 provider). Dropping this
+   also retires the headless token-freshness problem — manual weekly refresh
+   is fine.
+
+**Active:**
+9. ~~Surface remaining token validity on `doctor`~~ — **IMPLEMENTED 2026-06-28**
+   via the `/api/auth/session` `error` field (not `expires`). See §9.
+10. ~~Provider API-drift detection~~ — **IMPLEMENTED 2026-06-28** as the
+    `canary` command. See §10.
 
 **Deprioritized** (entries kept at the bottom of this file; revisit on
 demand): `--force` flags, per-conversation cache reset, official export-ZIP
-fallback, o1/o3 reasoning reclassification, Obsidian output, token expiry
-notifications (largely superseded by cookie auto-extraction), search command.
+fallback, o1/o3 reasoning reclassification, Obsidian output, search command.
 Additional web providers (Gemini/Grok/Perplexity) are explicitly out of
 scope — no significant usage to archive.
 
@@ -50,8 +65,8 @@ from live recon: the retrieval dumps are NOT flagged
 `is_visually_hidden_from_conversation` — `author.name == "file_search"` is
 the discriminator (the hidden flag only marks Custom Instructions and small
 system stubs). Verified live: worst files shrink 93% (524KB → 36KB).
-Remaining step: full-archive re-export (`cache --clear` + `export`) +
-Joplin re-sync, at the user's chosen time.
+Full-archive re-export (the `export --force` campaign) + Joplin re-sync
+completed 2026-06-13.
 
 **Problem (measured 2026-06-12 against a full fresh export):** 45% of the
 entire 11.2MB archive (260 files) is tool-role messages; 29 files are
@@ -89,37 +104,38 @@ file contains zero hidden-context blocks.
 Re-export workflow after shipping: `cache --clear` + `export` (same as
 the v0.4.0 migration).
 
-## 2. Brave Cookie Auto-Extraction — SHIPPED v0.6.0
+## 2. Brave Cookie Auto-Extraction — REMOVED (not viable)
 
-**Implemented 2026-06-12**: `auth --from-browser [browser]`
-(`src/browser_tokens.py`, browser-cookie3 dependency, defaults to brave;
-chrome/chromium/edge/firefox also supported). Extracted tokens are validated
-against the live API before `.env` is touched. **Discovered during
-implementation: Brave is not installed on this machine** — the logged-in
-browser lives elsewhere, so the flag only helps when the exporter runs on
-that machine (verified the graceful-failure path here; the happy path is
-covered by mocked tests). This strengthens the case for the StartOS
-token-push mechanism (entry 8).
+**Shipped v0.6.0 (2026-06-12), removed 2026-06-27.** `auth --from-browser`
+plus `src/browser_tokens.py` and the `browser-cookie3` dependency are gone.
+Auth is manual (DevTools) only. Do not re-attempt without a fundamentally
+different mechanism (see below).
 
-Pull session tokens directly from the Brave browser profile instead of the
-weekly manual DevTools dance. `auth` gains an "extract from browser" path
-(with the manual flow kept as fallback); a later iteration could let
-`doctor` or a 401 handler suggest/perform a re-extract automatically.
+**Why it doesn't work.** Modern Chromium browsers encrypt cookies on Windows
+with **App-Bound Encryption** (Chrome 127+, July 2024; current Brave).
+Cookies are written with a `v20` prefix and keyed off a secret wrapped in a
+**SYSTEM-level** DPAPI layer plus app validation. `browser-cookie3` only
+knows the legacy `v10`/DPAPI key, so its AES-GCM MAC check fails — the exact
+symptom hit in the field:
 
-- Brave on Linux follows the Chromium pattern: cookies SQLite at
-  `~/.config/BraveSoftware/Brave-Browser/Default/Cookies`, values
-  AES-128-CBC encrypted with a key derived from the OS keyring secret
-  ("Brave Safe Storage" via SecretService/kwallet, or the `peanuts` v10
-  fallback). Well-trodden territory — evaluate a small dependency
-  (`browser_cookie3`-style) vs. a focused in-repo implementation.
-- Extract `__Secure-next-auth.session-token.0` / `.1` from `chatgpt.com`
-  and `sessionKey` from `claude.ai`; write to `.env` through the existing
-  auth-wizard write path (tokens never echoed).
-- Brave must be the user's logged-in browser (it is); detect a locked /
-  unreadable cookie DB (Brave running with the DB exclusively locked) and
-  fall back to the manual flow with a clear message.
-- Does **not** solve headless/StartOS — no browser on the server. That
-  needs a token-push mechanism, tracked under the StartOS entry.
+```
+ChatGPT: Could not read brave cookies for chatgpt.com: Unable to get key for cookie decryption.
+```
+
+Decrypting `v20` at all requires unwrapping the SYSTEM layer, which means
+running as SYSTEM (e.g. a PsExec-style service) — i.e. **Administrator
+rights** and behavior that AV/EDR flags as infostealer activity. The one
+maintained Python option (`rookiepy`) needs admin from Chrome v130+, was
+**archived 2026-06-07**, and has an unresolved bug where **Brave returns 0
+cookies** even after the key is retrieved. ABE is *designed* to stop exactly
+this, so no off-disk reader is a reliable, non-invasive fit.
+
+**If ever revisited:** the only non-admin path is Chrome Remote Debugging
+(launch the browser with `--remote-debugging-port`, read cookies via
+`Network.getAllCookies` — the running browser decrypts for you). Heavier and
+intrusive; not worth it for a weekly token refresh that takes 30 seconds by
+hand. With the headless/StartOS direction dropped (#8), manual DevTools
+refresh is the accepted approach — no automated extraction is needed.
 
 ## 3. Claude Code Session Provider — SHIPPED v0.6.0
 
@@ -257,43 +273,174 @@ Note: the conversation *listing* (paginated, 100/page) still runs in full
 each time so the cache comparison works — the cap applies to the heavy
 per-conversation detail fetches, which dominate request volume.
 
-Together with watch mode, this is a stepping stone to the StartOS service:
-a scheduled, capped, politely-paced export is the traffic profile a
-headless deployment needs.
+This is a stepping stone to the StartOS service: a capped, politely-paced
+export — scheduled by the host (cron/StartOS), not an in-app loop — is the
+traffic profile a headless deployment needs.
 
-## 7. Scheduled / Watch Mode
+## 7. Scheduled / Watch Mode — DROPPED (2026-06-28)
 
-Add a `watch` command (or cron integration helper) to run exports automatically
-on a schedule:
+An in-app `watch`/scheduler loop is not worth building. Scheduling belongs to
+whatever hosts the tool: a user cron line locally, and on the long-term
+StartOS target the platform's own scheduling. Either way the tool only needs
+to do one capped, politely-paced `export` + `joplin` run and exit — which it
+already does. If cron ergonomics ever feel clunky, a thin `sync` subcommand
+that chains `export` then `joplin` for a single cron line is a trivial
+add-on, but the polling loop itself is off the roadmap.
 
-```bash
-python -m src.main watch --interval 6h   # poll every 6 hours
-```
+## 8. StartOS Service Packaging — DROPPED (2026-06-28)
 
-This would run `export` + `joplin` in sequence, then sleep. Alternatively,
-provide a `cron` command that prints the correct crontab line for the user's
-setup.
+Not pursuing a headless StartOS service. The local, manually-run CLI is
+sufficient: the source conversations live in the providers' clouds and can be
+re-downloaded, and Joplin already syncs (encrypted) to an offsite S3 provider,
+so durability is covered without a server in the loop.
 
-Implementation: simple loop with `time.sleep()`, or emit a crontab entry
-string that calls the export and joplin commands in sequence. A `--once`
-flag would do a single run then exit (useful for cron itself).
+Dropping this also retires the one genuinely hard sub-problem it carried —
+session-token freshness without a browser. There is no headless context to
+keep fresh; the weekly manual DevTools refresh is acceptable. (Local cookie
+extraction remains a dead end regardless — see #2.)
 
-Stepping stone to the StartOS service (below).
+## 9. Token Validity on `doctor` — IMPLEMENTED (2026-06-28)
 
-## 8. StartOS Service Packaging (long-term destination)
+Shipped: `doctor` now adds a "ChatGPT token active" check via
+`ChatGPTProvider.session_health()` (reads `/api/auth/session`, passes iff
+`error` is falsy and `accessToken` is present), the never-working JWE/`exp`
+decode path was removed, and `_fetch_access_token` now fails fast on a set
+`error` instead of returning a stale token. Tests in
+`tests/test_providers.py::TestChatGPTSessionHealth`. Investigation trail
+below for the record.
 
-Run the exporter headless as a StartOS service: scheduled export + sync to
-a Joplin Server instance, so the backup happens without a desktop in the
-loop.
+Goal: if it's cheap to tell how much longer a token will work, show it on
+`doctor`. Findings from live recon:
 
-- Building blocks needed first: watch mode (#7), session limiter (#6),
-  and a Joplin Server (vs. desktop Web Clipper) sync target.
-- **Hard problem, flagged early:** session-token freshness without a
-  browser. ChatGPT tokens last ~7 days; there is no browser on the server
-  to re-extract from. Candidate mechanisms: a service config action the
-  user pastes fresh tokens into periodically, or a future browser
-  extension that pushes tokens to the server. Cookie auto-extraction (#2)
-  only solves the desktop case.
+- **Not readable from the token itself.** ChatGPT's `CHATGPT_SESSION_TOKEN`
+  is a **JWE** (header `{"alg":"dir","enc":"A256GCM"}`, `eyJ…` prefix is just
+  the encrypted protected header) — the `exp` claim is AES-256-GCM encrypted
+  with an OpenAI-only key, so it cannot be decoded client-side. Claude's
+  `sk-…` key is fully opaque. The existing `doctor` JWT-decode path therefore
+  never yields an expiry for the real tokens (falls to the "not decodable"
+  branch).
+- **`/api/auth/session` exposes an `expires`** (the provider already calls
+  this endpoint in `_fetch_access_token`; the response includes `expires`
+  alongside `accessToken`). Live value observed 2026-06-28:
+  `2026-09-26` — **~90 days out**. This contradicts both the code's ~7-day
+  assumption and the lived weekly-refresh cadence, so it is almost certainly
+  the NextAuth **rolling session window** (re-extended on every call), not
+  the point at which the pasted token actually 401s. Displaying it verbatim
+  would give false confidence.
+- **RESOLVED 2026-06-28 by a live 401 data point.** When the ChatGPT token
+  was actually dead (conversations API → 401), `/api/auth/session` still
+  returned **HTTP 200** with `expires: 2026-09-26` (~90 days out) — and that
+  `expires` *advanced* between two calls seconds apart (`04:20:08` → `04:30:37`).
+  So `expires` is a **rolling session window that rolls forward on every call
+  even for a dead token**; displaying it would actively lie. The same response
+  carried `error: "RefreshAccessTokenError"` and a stale `accessToken`.
+- **The real signal is `error`, not `expires` or `accessToken`.** On a healthy
+  token `error` is absent/null; when the session token is dead NextAuth can't
+  refresh and sets `error: "RefreshAccessTokenError"` while still echoing a
+  rolling `expires` and a stale `accessToken`. This is an exact, free, binary
+  health check.
+- **Design (ready to build):**
+  1. `doctor` ChatGPT check → read `/api/auth/session`; pass iff `error` is
+     falsy and `accessToken` present; on `RefreshAccessTokenError` report
+     "token expired — refresh". Drop the JWE/`exp` decode path (it can never
+     work) and do NOT surface `expires`.
+  2. Latent bug to fix alongside: `_fetch_access_token` reads `accessToken`
+     without checking `error`, so it proceeds with a stale token and yields a
+     confusing downstream 401 instead of a clear "refresh your token" message.
+     Check `error` there and fail fast.
+  3. Claude stays a 401-only signal (opaque `sk-`, no equivalent endpoint).
+
+## 10. Provider API-Drift Detection — IMPLEMENTED (2026-06-28)
+
+Shipped: the `canary` command + `BaseProvider.check_drift()` (overridden by
+ChatGPT and Claude). It fetches one listing page + one conversation per
+provider and asserts only the normalizer's load-bearing fields, emitting
+`DRIFT_OK/WARN/ERROR` findings (`src/providers/base.py`). Severity badges
+print as a Rich table; ERROR exits non-zero, WARN is non-fatal so a backup
+run is never blocked. Drift vocabularies (`_KNOWN_TOOL_AUTHORS`,
+`_HANDLED_CONTENT_TYPES`) live in `chatgpt.py` next to the collapse set they
+guard. Tests: `TestChatGPTDriftCanary`, `TestClaudeDriftCanary`,
+`TestCanaryCommand`. Verified live 2026-06-28 — both providers OK. Recon
+trail below for the record.
+
+## 10b. Provider API-Drift Detection — investigation (2026-06-28)
+
+The export depends on undocumented internal web APIs (ChatGPT/Claude) that can
+change shape without notice. The worst failure for a backup tool is *silent*:
+a response-schema change that makes the exporter skip or mis-parse content
+without erroring. `doctor` currently checks token validity, reachability, and
+manifest↔disk integrity — but not "does the provider's response still look
+like what the parser expects."
+
+To investigate: a lightweight schema/shape assertion on a known-good sample
+of each provider's listing + conversation-detail responses (presence and type
+of the fields the normalizers rely on), surfaced as a `doctor` check or a
+dedicated canary.
+
+**Live recon — Claude captured 2026-06-28 (ChatGPT pending a token refresh):**
+
+- **Dependency surface (assert ONLY these — see below for why):**
+  - listing item: `uuid`, `name`, `updated_at`/`created_at`, `project.name`.
+  - conversation detail: `uuid`/`id`, `name`, `created_at`, `updated_at`,
+    `project.name`, `chat_messages[]`.
+  - message: `sender` (`human`/`assistant`), `text` (string) or `content`
+    (list of typed blocks), `created_at`.
+- **Key finding — full-shape diffing is the wrong design.** Claude's
+  `settings` object is full of volatile internal codenames that churn
+  constantly: `enabled_bananagrams`, `enabled_sourdough`, `enabled_foccacia`,
+  `enabled_saffron`, `enabled_turmeric`, `enabled_monkeys_in_a_barrel`,
+  `paprika_mode`, `enabled_megaminds`, … A "any new/removed key = drift"
+  canary would fire on every UI experiment. The canary MUST target the
+  normalizer's load-bearing fields only, not the whole response. (Aligns with
+  the drop-noise-don't-retain-it principle.)
+- **Real Claude messages are flat `text`/`sender`** — in this archive every
+  message had a string `text` and NO `content` block list (0 rich blocks
+  observed). So `_extract_claude_blocks` / `_dispatch_claude_block` (tool_use,
+  thinking, image, …) is an **unexercised theoretical path**; drift there
+  can't be "caught" by a canary because it never runs on real data — it's a
+  safety net for if Claude ever switches to block content. The canary should
+  assert the flat shape and *warn if `content` ever appears as a list* (that
+  itself is the drift event that would activate the dormant code).
+- **Possible silent-loss spot (separate from drift):** Claude messages carry
+  `attachments` and `files` arrays (empty in this sample) that the normalizer
+  ignores entirely. If a user ever attaches files in Claude, they'd be
+  dropped without a LossReport entry. Worth a follow-up check.
+**Live recon — ChatGPT captured 2026-06-28:**
+
+- **Dependency surface (assert ONLY these):**
+  - listing item: `id`, `title`, `update_time`/`create_time`.
+  - conversation detail: `conversation_id`/`id`, `title`, `create_time`,
+    `update_time`, `mapping` (non-empty).
+  - mapping node: `message`, `children` (the tree walk depends on both);
+    message: `author.role`, `author.name`, `content.content_type`,
+    `content.parts`, `metadata.is_visually_hidden_from_conversation`.
+- **content_type vocabulary observed (all currently handled):** `text`,
+  `model_editable_context`, `multimodal_text`, `thoughts`, `code`,
+  `execution_output`, `reasoning_recap`, `user_editable_context`,
+  `tether_browsing_display`. A *new* content_type already degrades gracefully
+  (visible `unknown` block + WARNING + LossReport tally) — so content_type
+  drift is **already non-silent**. The canary just needs to confirm the known
+  set still parses to non-empty blocks.
+- **The genuinely silent drift risk — `author.name` collapse keys.**
+  `_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}`. Recon
+  confirms `file_search` is live (and `web.run`/`python` are correctly left
+  un-collapsed). If OpenAI renames `file_search`, the collapse **silently
+  stops** and the archive re-bloats with no error or LossReport entry. This is
+  the top canary target: assert that retrieval-dump tool authors are still
+  recognized, or at least flag unfamiliar `(role="tool", author.name)` pairs.
+- **Second silent risk — empty `content.parts`.** A `text` message whose
+  `parts` field is renamed/emptied yields zero blocks and is skipped with only
+  a debug log = silent loss. Canary should assert a sampled `text` message
+  produces a non-empty block.
+
+**Recon complete for both providers. Canary design (ready to build):** a
+`doctor` check (or dedicated `canary` command) that, per provider, fetches one
+listing page + one conversation and asserts the dependency-surface fields
+above by presence+type — NOT full shape (Claude `settings` codenames prove
+full-shape diffing is pure noise). Specific tripwires: (ChatGPT) unfamiliar
+`(tool, author.name)` pair and empty `parts` on a text message; (Claude)
+`content` appearing as a list, and non-empty `attachments`/`files`. Failures
+surface as a warning, never a hard error (a backup tool must still run).
 
 ---
 
@@ -369,16 +516,13 @@ structure matching the user's Obsidian setup. No API needed — just file I/O.
 
 ## Token Expiry Notifications
 
-Proactively warn when a token is close to expiry (within 48h for ChatGPT),
-rather than only surfacing the warning at startup. Options:
-
-- Add an `expiry` subcommand that prints token status and exits non-zero if
-  any token is expired or expiring soon (useful in scripts/cron)
-- Send a desktop notification via `notify-send` (Linux) or `osascript` (macOS)
-  when a token is within 24h of expiry
-
-Largely superseded by Brave cookie auto-extraction (#2): once refresh is
-one command (or automatic), expiry warnings matter much less.
+Moved to the active roadmap as §9 (Token Validity on `doctor`). The original
+"proactively notify before expiry" idea is blocked by the same finding: a
+reliable expiry time isn't available client-side (ChatGPT token is encrypted,
+Claude's is opaque, and `/api/auth/session`'s `expires` looks like a rolling
+window rather than the real refresh cadence). Any heads-up — a `doctor`
+line, an `expiry` subcommand, or a `notify-send` nudge — depends on first
+resolving the §9 open question of what signal is actually trustworthy.
 
 ## Search Command
 

README.md

@@ -79,7 +79,17 @@ Before anything else, validate your setup:
 ai-chat-exporter doctor
 ```
 
-This checks token presence, format, expiry, directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.
+This checks token presence, format, token health (via the `/api/auth/session` `error` field), directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.
+
+### Checking for API drift
+
+The exporter reads ChatGPT's and Claude's undocumented internal web APIs, which can change shape without notice. The worst failure for a backup tool is *silent* — a response change that makes the exporter skip or mis-parse content without erroring. Run the canary to catch that early:
+
+```bash
+ai-chat-exporter canary
+```
+
+It probes one conversation per provider and checks only the fields the parser depends on (a renamed retrieval-tool author that would bypass the content-collapse, a new content type, drifted message fields, ignored attachments). `OK` means the live schema still matches; `WARN`/`ERROR` tells you exactly what changed. Worth running before a large export or on a schedule.
 
 ---
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ai-chat-exporter"
-version = "0.6.0"
+version = "0.7.0"
 description = "Export ChatGPT and Claude conversation history to Markdown for personal archival in Joplin"
 requires-python = ">=3.11"
 dependencies = [

src/main.py

@@ -18,7 +18,7 @@ from src.cache import Cache, CacheError
 from src.config import ConfigError
 from src.logging_config import setup_logging
 from src.loss_report import LossReport
-from src.providers.base import ProviderError
+from src.providers.base import DRIFT_ERROR, DRIFT_OK, DRIFT_WARN, ProviderError
 
 console = Console()
 err_console = Console(stderr=True)
@@ -332,6 +332,91 @@ def doctor(ctx: click.Context) -> None:
         sys.exit(1)
 
 
+@cli.command()
+@click.option(
+    "--provider",
+    type=click.Choice(["chatgpt", "claude", "all"]),
+    default="all",
+    help="Which web-API provider(s) to probe.",
+)
+@click.pass_context
+def canary(ctx: click.Context, provider: str) -> None:
+    """Probe provider APIs for schema drift against the fields the parser needs.
+
+    Fetches one listing page + one conversation per provider and checks only
+    the normalizer's load-bearing fields (not the full response shape, which
+    churns harmlessly). Surfaces the silent-failure risks for a backup tool:
+    a renamed retrieval-tool author bypassing the hidden-content collapse, a
+    new content_type, drifted message fields, or ignored attachments.
+
+    Exits non-zero if any ERROR finding is reported (a load-bearing field is
+    missing/mistyped). WARN findings are surfaced but do not fail. See
+    FUTURE.md §10.
+    """
+    from dotenv import load_dotenv
+
+    load_dotenv(override=False)
+
+    chatgpt_token = os.getenv("CHATGPT_SESSION_TOKEN", "").strip() or None
+    claude_key = os.getenv("CLAUDE_SESSION_KEY", "").strip() or None
+
+    targets: list[tuple[str, object]] = []
+    if provider in ("chatgpt", "all") and chatgpt_token:
+        from src.providers.chatgpt import ChatGPTProvider
+
+        ct1 = os.getenv("CHATGPT_SESSION_TOKEN_1", "").strip() or None
+        targets.append(
+            ("chatgpt", lambda: ChatGPTProvider(chatgpt_token, session_token_1=ct1))
+        )
+    if provider in ("claude", "all") and claude_key:
+        from src.providers.claude import ClaudeProvider
+
+        targets.append(("claude", lambda: ClaudeProvider(claude_key)))
+
+    if not targets:
+        err_console.print(
+            "[yellow]No web-API provider tokens configured "
+            "(set CHATGPT_SESSION_TOKEN / CLAUDE_SESSION_KEY).[/yellow]"
+        )
+        sys.exit(1)
+
+    rows: list[tuple[str, dict]] = []
+    for name, factory in targets:
+        try:
+            findings = factory().check_drift()
+        except ProviderError as e:
+            findings = [{"severity": DRIFT_ERROR, "check": "probe", "detail": str(e.original)[:100]}]
+        except Exception as e:
+            findings = [{"severity": DRIFT_ERROR, "check": "probe", "detail": str(e)[:100]}]
+        for f in findings:
+            rows.append((name, f))
+
+    _print_canary_table(rows)
+    if any(f["severity"] == DRIFT_ERROR for _, f in rows):
+        sys.exit(1)
+
+
+def _print_canary_table(rows: list[tuple[str, dict]]) -> None:
+    table = Table(title="API Drift Canary", show_header=True)
+    table.add_column("Provider", style="bold")
+    table.add_column("Severity", justify="center")
+    table.add_column("Check")
+    table.add_column("Detail")
+    badge = {
+        DRIFT_OK: "[green]OK[/green]",
+        DRIFT_WARN: "[yellow]WARN[/yellow]",
+        DRIFT_ERROR: "[red]ERROR[/red]",
+    }
+    for name, f in rows:
+        table.add_row(
+            name,
+            badge.get(f["severity"], f["severity"]),
+            f.get("check", ""),
+            f.get("detail", ""),
+        )
+    console.print(table)
+
+
 def _run_doctor_checks(cache: Cache | None = None) -> list[dict]:
     """Run all doctor checks and return results.
 

src/providers/base.py

@@ -50,6 +50,27 @@ VALID_HIDDEN_CONTENT_POLICIES = {
     HIDDEN_CONTENT_OMIT,
 }
 
+# ---------------------------------------------------------------------------
+# API-drift canary (FUTURE.md §10)
+# ---------------------------------------------------------------------------
+# The export depends on undocumented internal web APIs that can change shape
+# without notice. The canary fetches a small live sample and asserts only the
+# normalizer's load-bearing fields (NOT full shape — provider settings objects
+# churn constantly and would be pure noise). Findings carry a severity:
+#   ERROR — a load-bearing field is missing/mistyped; the parser will break or
+#           silently lose data. The backup is no longer trustworthy.
+#   WARN  — something unfamiliar appeared (new content_type, new tool author,
+#           a dormant field went live). Investigate; not necessarily broken.
+#   OK    — the assertion held.
+DRIFT_ERROR = "error"
+DRIFT_WARN = "warn"
+DRIFT_OK = "ok"
+
+
+def drift_finding(severity: str, check: str, detail: str = "") -> dict:
+    """Build one canary finding."""
+    return {"severity": severity, "check": check, "detail": detail}
+
 
 def resolve_hidden_content_policy() -> str:
     """Read EXPORTER_HIDDEN_CONTENT from the environment, defaulting to placeholder."""
@@ -159,6 +180,15 @@ class BaseProvider(ABC):
         isolation, e.g. from tests, doesn't crash).
         """
 
+    def check_drift(self) -> list[dict]:
+        """Probe the live API and assert the normalizer's load-bearing fields.
+
+        Returns a list of ``drift_finding`` dicts. The default is a no-op
+        (used by providers with no remote schema risk, e.g. local Claude Code
+        transcripts); web-API providers override this. See FUTURE.md §10.
+        """
+        return []
+
     # ------------------------------------------------------------------
     # Concrete helpers
     # ------------------------------------------------------------------

src/providers/chatgpt.py

@@ -45,12 +45,16 @@ from src.blocks import (
 from src.loss_report import LossReport
 from src.providers.base import (
     BaseProvider,
+    DRIFT_ERROR,
+    DRIFT_OK,
+    DRIFT_WARN,
     HIDDEN_CONTENT_FULL,
     HIDDEN_CONTENT_OMIT,
     HIDDEN_CONTENT_PLACEHOLDER,
     ProviderError,
     REQUEST_TIMEOUT,
     VALID_HIDDEN_CONTENT_POLICIES,
+    drift_finding,
     resolve_hidden_content_policy,
 )
 
@@ -91,6 +95,20 @@ def parse_asset_file_id(ref: str) -> str | None:
 # myfiles_browser is the legacy name for the same retrieval tool.
 _COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}
 
+# ── API-drift canary vocabularies (FUTURE.md §10) ──────────────────────────
+# Tool authors seen live 2026-06-28. The collapse keys off author.name, so a
+# RENAMED retrieval tool would silently bypass the collapse and re-bloat the
+# archive — the canary flags any unfamiliar (role="tool", author.name) pair.
+_KNOWN_TOOL_AUTHORS = _COLLAPSE_TOOL_AUTHORS | {"web.run", "python"}
+# content_types the normalizer dispatches on (keep in sync with
+# _extract_blocks_for_content). A new value already degrades to an `unknown`
+# block + LossReport at export time; the canary flags it proactively.
+_HANDLED_CONTENT_TYPES = frozenset({
+    "text", "multimodal_text", "execution_output", "system_error",
+    "tether_browsing_display", "code", "thoughts", "reasoning_recap",
+    "user_editable_context", "model_editable_context", "image_asset_pointer",
+})
+
 
 class ChatGPTProvider(BaseProvider):
     """Provider for ChatGPT conversations via the internal web API.
@@ -766,6 +784,84 @@ class ChatGPTProvider(BaseProvider):
             "messages": messages,
         }
 
+    def check_drift(self) -> list[dict]:
+        """Probe one listing page + one conversation; assert load-bearing fields.
+
+        See FUTURE.md §10. Asserts only the fields the normalizer depends on —
+        not the full response shape (which churns harmlessly). The top silent
+        risk is a renamed retrieval tool author bypassing the collapse.
+        """
+        findings: list[dict] = []
+        page = self.list_conversations(offset=0, limit=5)
+        if not page:
+            return [drift_finding(DRIFT_WARN, "listing",
+                                  "empty listing — cannot verify shape")]
+
+        item = page[0]
+        for f in ("id", "title"):
+            if f not in item:
+                findings.append(drift_finding(
+                    DRIFT_ERROR, "listing", f"summary item missing '{f}'"))
+        if not (item.get("update_time") or item.get("create_time")):
+            findings.append(drift_finding(
+                DRIFT_WARN, "listing", "no update_time/create_time on summary"))
+
+        conv_id = item.get("id")
+        if not conv_id:
+            return findings or [drift_finding(DRIFT_ERROR, "listing", "no id to fetch")]
+        raw = self.get_conversation(conv_id)
+        if not (raw.get("conversation_id") or raw.get("id")):
+            findings.append(drift_finding(
+                DRIFT_ERROR, "detail", "no conversation_id/id on detail"))
+        mapping = raw.get("mapping")
+        if not isinstance(mapping, dict) or not mapping:
+            findings.append(drift_finding(
+                DRIFT_ERROR, "detail", "mapping missing/empty — tree walk will yield 0 messages"))
+            return findings
+
+        saw_text_with_parts = False
+        seen_new_ct: set[str] = set()
+        seen_new_tool: set[str] = set()
+        for node in mapping.values():
+            if "children" not in node:
+                findings.append(drift_finding(
+                    DRIFT_ERROR, "mapping", "node missing 'children' — tree walk breaks"))
+                break
+            msg = node.get("message")
+            if not msg:
+                continue
+            author = msg.get("author") or {}
+            role, name = author.get("role"), author.get("name")
+            content = msg.get("content") or {}
+            ct = content.get("content_type")
+            if ct and ct not in _HANDLED_CONTENT_TYPES:
+                seen_new_ct.add(ct)
+            if role == "tool" and name and name not in _KNOWN_TOOL_AUTHORS:
+                seen_new_tool.add(name)
+            if ct == "text":
+                parts = content.get("parts") or []
+                if any(isinstance(p, str) and p.strip() for p in parts):
+                    saw_text_with_parts = True
+
+        for name in sorted(seen_new_tool):
+            findings.append(drift_finding(
+                DRIFT_WARN, "collapse",
+                f"unfamiliar tool author {name!r} — if it's a retrieval dump it "
+                "is NOT being collapsed (silent archive bloat); update "
+                "_COLLAPSE_TOOL_AUTHORS / _KNOWN_TOOL_AUTHORS"))
+        for ct in sorted(seen_new_ct):
+            findings.append(drift_finding(
+                DRIFT_WARN, "content_type",
+                f"new content_type {ct!r} — exported as an unknown block; add a handler"))
+        if not saw_text_with_parts:
+            findings.append(drift_finding(
+                DRIFT_WARN, "parts",
+                "no text message yielded non-empty parts — content.parts may have drifted"))
+
+        if not findings:
+            findings.append(drift_finding(DRIFT_OK, "schema", "all load-bearing fields present"))
+        return findings
+
 
 # ---------------------------------------------------------------------------
 # Internal helpers

src/providers/claude.py

@@ -16,7 +16,14 @@ from src.blocks import (
     make_unknown_block,
 )
 from src.loss_report import LossReport
-from src.providers.base import BaseProvider, ProviderError
+from src.providers.base import (
+    BaseProvider,
+    DRIFT_ERROR,
+    DRIFT_OK,
+    DRIFT_WARN,
+    ProviderError,
+    drift_finding,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -232,6 +239,70 @@ class ClaudeProvider(BaseProvider):
             "messages": messages,
         }
 
+    def check_drift(self) -> list[dict]:
+        """Probe one listing page + one conversation; assert load-bearing fields.
+
+        See FUTURE.md §10. Real Claude messages are flat ``text``/``sender``;
+        the rich ``content`` block path is dormant, so its activation (content
+        becoming a list) is itself a drift event worth flagging. ``attachments``
+        / ``files`` are ignored by the normalizer — non-empty values are a
+        silent-loss risk.
+        """
+        findings: list[dict] = []
+        page = self.list_conversations(offset=0, limit=5)
+        if not page:
+            return [drift_finding(DRIFT_WARN, "listing",
+                                  "empty listing — cannot verify shape")]
+
+        item = page[0]
+        if not (item.get("uuid") or item.get("id")):
+            findings.append(drift_finding(DRIFT_ERROR, "listing", "summary missing uuid/id"))
+        if not (item.get("name") or item.get("title")):
+            findings.append(drift_finding(DRIFT_WARN, "listing", "no name/title on summary"))
+        if not (item.get("updated_at") or item.get("update_time")):
+            findings.append(drift_finding(DRIFT_WARN, "listing", "no updated_at on summary"))
+
+        conv_id = item.get("uuid") or item.get("id")
+        if not conv_id:
+            return findings or [drift_finding(DRIFT_ERROR, "listing", "no id to fetch")]
+        raw = self.get_conversation(conv_id)
+        if not (raw.get("uuid") or raw.get("id")):
+            findings.append(drift_finding(DRIFT_ERROR, "detail", "no uuid/id on detail"))
+        msgs = raw.get("chat_messages") or raw.get("messages")
+        if not isinstance(msgs, list) or not msgs:
+            findings.append(drift_finding(
+                DRIFT_ERROR, "detail", "chat_messages missing/empty — 0 messages"))
+            return findings
+
+        saw_sender = saw_text = False
+        flagged_list = flagged_attach = False
+        for m in msgs:
+            if m.get("sender") or m.get("role"):
+                saw_sender = True
+            content = m.get("content")
+            if isinstance(content, list) and not flagged_list:
+                findings.append(drift_finding(
+                    DRIFT_WARN, "content",
+                    "message.content is now a LIST — Claude switched to typed blocks; "
+                    "the dormant _dispatch_claude_block path is now live, verify rendering"))
+                flagged_list = True
+            if m.get("text") or content:
+                saw_text = True
+            if (m.get("attachments") or m.get("files")) and not flagged_attach:
+                findings.append(drift_finding(
+                    DRIFT_WARN, "attachments",
+                    "message has non-empty attachments/files — the normalizer ignores "
+                    "these (silent loss); add handling"))
+                flagged_attach = True
+
+        if not saw_sender:
+            findings.append(drift_finding(DRIFT_ERROR, "messages", "no sender/role on any message"))
+        if not saw_text:
+            findings.append(drift_finding(DRIFT_WARN, "messages", "no text/content found"))
+        if not findings:
+            findings.append(drift_finding(DRIFT_OK, "schema", "all load-bearing fields present"))
+        return findings
+
 
 # ---------------------------------------------------------------------------
 # Internal helpers

tests/test_cli.py

@@ -275,3 +275,25 @@ class TestPrune:
         assert result.exit_code == 0
         assert "Aborted" in result.output
         assert stale.exists()
+
+
+class TestCanaryCommand:
+    """`canary` wiring — no live API calls (no tokens configured)."""
+
+    def test_no_tokens_exits_nonzero(self, tmp_path):
+        cache = Cache(tmp_path / "cache")
+        cache.acknowledge_tos()
+        runner = CliRunner(mix_stderr=True)
+        result = runner.invoke(
+            cli,
+            ["--no-log-file", "canary"],
+            env={
+                "CACHE_DIR": str(tmp_path / "cache"),
+                "EXPORT_DIR": str(tmp_path / "exports"),
+                # Empty so .env (override=False) can't repopulate real tokens.
+                "CHATGPT_SESSION_TOKEN": "",
+                "CLAUDE_SESSION_KEY": "",
+            },
+        )
+        assert result.exit_code == 1
+        assert "No web-API provider tokens" in result.output

tests/test_providers.py

@@ -973,3 +973,146 @@ class TestChatGPTSessionHealth:
         ok, detail = p.session_health()
         assert ok is False
         assert "unreachable" in detail.lower()
+
+
+# ---------------------------------------------------------------------------
+# API-drift canary (§10): assert load-bearing fields, flag silent risks.
+# ---------------------------------------------------------------------------
+
+
+def _sev(findings):
+    return {f["severity"] for f in findings}
+
+
+def _checks(findings, severity):
+    return {f["check"] for f in findings if f["severity"] == severity}
+
+
+class TestChatGPTDriftCanary:
+    def _provider(self, page, detail):
+        from src.providers.chatgpt import ChatGPTProvider
+
+        p = ChatGPTProvider.__new__(ChatGPTProvider)
+        p.list_conversations = lambda offset=0, limit=5: page
+        p.get_conversation = lambda cid: detail
+        return p
+
+    @staticmethod
+    def _detail(mapping):
+        return {
+            "conversation_id": "c1", "title": "T",
+            "create_time": 1.0, "update_time": 2.0, "mapping": mapping,
+        }
+
+    @staticmethod
+    def _node(role="user", name=None, content_type="text", parts=("hello",)):
+        return {
+            "id": "n", "parent": None, "children": [],
+            "message": {
+                "author": {"role": role, "name": name},
+                "content": {"content_type": content_type, "parts": list(parts)},
+                "metadata": {},
+            },
+        }
+
+    def test_healthy_shape_is_ok_only(self):
+        from src.providers.base import DRIFT_OK
+
+        p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
+                           self._detail({"n1": self._node()}))
+        assert _sev(p.check_drift()) == {DRIFT_OK}
+
+    def test_missing_listing_id_is_error(self):
+        from src.providers.base import DRIFT_ERROR
+
+        p = self._provider([{"title": "T"}], self._detail({"n1": self._node()}))
+        assert DRIFT_ERROR in _sev(p.check_drift())
+
+    def test_empty_mapping_is_error(self):
+        from src.providers.base import DRIFT_ERROR
+
+        p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
+                           self._detail({}))
+        findings = p.check_drift()
+        assert "detail" in _checks(findings, DRIFT_ERROR)
+
+    def test_unfamiliar_tool_author_warns_collapse(self):
+        from src.providers.base import DRIFT_WARN
+
+        mapping = {
+            "n1": self._node(),  # a healthy text msg so parts check passes
+            "n2": self._node(role="tool", name="brand_new_retriever"),
+        }
+        p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
+                           self._detail(mapping))
+        assert "collapse" in _checks(p.check_drift(), DRIFT_WARN)
+
+    def test_new_content_type_warns(self):
+        from src.providers.base import DRIFT_WARN
+
+        mapping = {
+            "n1": self._node(),
+            "n2": self._node(content_type="hologram_v2", parts=()),
+        }
+        p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
+                           self._detail(mapping))
+        assert "content_type" in _checks(p.check_drift(), DRIFT_WARN)
+
+    def test_known_tool_author_not_flagged(self):
+        # file_search is a known collapse author — must NOT warn.
+        mapping = {
+            "n1": self._node(),
+            "n2": self._node(role="tool", name="file_search"),
+        }
+        p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
+                           self._detail(mapping))
+        assert "collapse" not in _checks(p.check_drift(), "warn")
+
+
+class TestClaudeDriftCanary:
+    def _provider(self, page, detail):
+        from src.providers.claude import ClaudeProvider
+
+        p = ClaudeProvider.__new__(ClaudeProvider)
+        p.list_conversations = lambda offset=0, limit=5: page
+        p.get_conversation = lambda cid: detail
+        return p
+
+    @staticmethod
+    def _detail(msgs):
+        return {"uuid": "u1", "name": "N", "created_at": "x", "updated_at": "y",
+                "chat_messages": msgs}
+
+    def test_healthy_flat_shape_is_ok(self):
+        from src.providers.base import DRIFT_OK
+
+        msgs = [{"uuid": "m1", "sender": "human", "text": "hi",
+                 "created_at": "x", "attachments": [], "files": []}]
+        p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
+                           self._detail(msgs))
+        assert _sev(p.check_drift()) == {DRIFT_OK}
+
+    def test_content_as_list_warns(self):
+        from src.providers.base import DRIFT_WARN
+
+        msgs = [{"uuid": "m1", "sender": "assistant",
+                 "content": [{"type": "text", "text": "hi"}], "created_at": "x"}]
+        p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
+                           self._detail(msgs))
+        assert "content" in _checks(p.check_drift(), DRIFT_WARN)
+
+    def test_nonempty_attachments_warn(self):
+        from src.providers.base import DRIFT_WARN
+
+        msgs = [{"uuid": "m1", "sender": "human", "text": "hi",
+                 "created_at": "x", "attachments": [{"file_name": "a.pdf"}]}]
+        p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
+                           self._detail(msgs))
+        assert "attachments" in _checks(p.check_drift(), DRIFT_WARN)
+
+    def test_empty_messages_is_error(self):
+        from src.providers.base import DRIFT_ERROR
+
+        p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
+                           self._detail([]))
+        assert DRIFT_ERROR in _sev(p.check_drift())