canary: detect provider API schema drift; release v0.7.0

The export reads ChatGPT's and Claude's undocumented internal web APIs,
which can change shape without notice; the worst failure for a backup tool
is a silent one (skipped/mis-parsed content with no error). Add a `canary`
command + BaseProvider.check_drift() (overridden by ChatGPT/Claude) that
fetches one listing page + one conversation per provider and asserts only
the normalizer's load-bearing fields — not the full response shape, which
churns harmlessly. The top silent risk it guards is a renamed retrieval-tool
author bypassing the hidden-content collapse. ERROR findings exit non-zero;
WARN findings are surfaced but non-fatal so a backup run is never blocked.

Also retires the in-app watch mode and headless StartOS direction from the
roadmap (tool stays a local, manually-run CLI) and updates docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

FUTURE.md

@@ -17,14 +17,18 @@ of these additions straightforward.
 
 Priorities reflect the tool's primary purpose — a trustworthy backup so that
 conversation data is not lost if a provider account is ever closed — plus the
-day-to-day friction of weekly ChatGPT token refresh, and the long-term goal
-of running headless as a StartOS service.
+day-to-day friction of the weekly ChatGPT token refresh. The tool stays a
+local, manually-run CLI; the headless/StartOS direction was dropped
+2026-06-28 (see #7 and #8), which also retires the token-freshness problem
+(manual refresh is sufficient).
 
 **Now (in order):**
 1. ~~Collapse tool retrieval dumps & hidden context~~ — **shipped in v0.6.0**
-   (full-archive re-export still pending; see entry below)
-2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0** (note: runs on
-   whichever machine hosts the browser — Brave is not on this Linux box)
+   (full-archive `export --force` re-export completed 2026-06-13)
+2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0, removed afterward.**
+   Not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave)
+   needs admin + SYSTEM impersonation that AV flags as credential theft, and
+   fails on Brave specifically. Auth is manual (DevTools) — see entry below.
 3. ~~Claude Code session provider~~ — **shipped in v0.6.0**
 4. ~~Archive hygiene — `prune` + `doctor` integrity check~~ — **shipped in v0.6.0**
 
@@ -32,12 +36,23 @@ of running headless as a StartOS service.
 
 5. ~~Binary content downloads~~ — **shipped in v0.6.0**
 6. ~~Per-session download limiter + polite pacing~~ — **shipped in v0.6.0**
-7. Scheduled / watch mode → StartOS service packaging (long-term destination)
+7. ~~Scheduled / watch mode~~ — **dropped 2026-06-28**; the tool stays a
+   manually-run CLI, so no in-app polling loop is needed
+8. ~~StartOS service packaging~~ — **dropped 2026-06-28**; the local CLI is
+   sufficient (source convos live in the cloud and can be re-downloaded;
+   Joplin already syncs encrypted to an offsite S3 provider). Dropping this
+   also retires the headless token-freshness problem — manual weekly refresh
+   is fine.
+
+**Active:**
+9. ~~Surface remaining token validity on `doctor`~~ — **IMPLEMENTED 2026-06-28**
+   via the `/api/auth/session` `error` field (not `expires`). See §9.
+10. ~~Provider API-drift detection~~ — **IMPLEMENTED 2026-06-28** as the
+    `canary` command. See §10.
 
 **Deprioritized** (entries kept at the bottom of this file; revisit on
 demand): `--force` flags, per-conversation cache reset, official export-ZIP
-fallback, o1/o3 reasoning reclassification, Obsidian output, token expiry
-notifications (largely superseded by cookie auto-extraction), search command.
+fallback, o1/o3 reasoning reclassification, Obsidian output, search command.
 Additional web providers (Gemini/Grok/Perplexity) are explicitly out of
 scope — no significant usage to archive.
 
@@ -50,8 +65,8 @@ from live recon: the retrieval dumps are NOT flagged
 `is_visually_hidden_from_conversation` — `author.name == "file_search"` is
 the discriminator (the hidden flag only marks Custom Instructions and small
 system stubs). Verified live: worst files shrink 93% (524KB → 36KB).
-Remaining step: full-archive re-export (`cache --clear` + `export`) +
-Joplin re-sync, at the user's chosen time.
+Full-archive re-export (the `export --force` campaign) + Joplin re-sync
+completed 2026-06-13.
 
 **Problem (measured 2026-06-12 against a full fresh export):** 45% of the
 entire 11.2MB archive (260 files) is tool-role messages; 29 files are
@@ -89,37 +104,38 @@ file contains zero hidden-context blocks.
 Re-export workflow after shipping: `cache --clear` + `export` (same as
 the v0.4.0 migration).
 
-## 2. Brave Cookie Auto-Extraction — SHIPPED v0.6.0
+## 2. Brave Cookie Auto-Extraction — REMOVED (not viable)
 
-**Implemented 2026-06-12**: `auth --from-browser [browser]`
-(`src/browser_tokens.py`, browser-cookie3 dependency, defaults to brave;
-chrome/chromium/edge/firefox also supported). Extracted tokens are validated
-against the live API before `.env` is touched. **Discovered during
-implementation: Brave is not installed on this machine** — the logged-in
-browser lives elsewhere, so the flag only helps when the exporter runs on
-that machine (verified the graceful-failure path here; the happy path is
-covered by mocked tests). This strengthens the case for the StartOS
-token-push mechanism (entry 8).
+**Shipped v0.6.0 (2026-06-12), removed 2026-06-27.** `auth --from-browser`
+plus `src/browser_tokens.py` and the `browser-cookie3` dependency are gone.
+Auth is manual (DevTools) only. Do not re-attempt without a fundamentally
+different mechanism (see below).
 
-Pull session tokens directly from the Brave browser profile instead of the
-weekly manual DevTools dance. `auth` gains an "extract from browser" path
-(with the manual flow kept as fallback); a later iteration could let
-`doctor` or a 401 handler suggest/perform a re-extract automatically.
+**Why it doesn't work.** Modern Chromium browsers encrypt cookies on Windows
+with **App-Bound Encryption** (Chrome 127+, July 2024; current Brave).
+Cookies are written with a `v20` prefix and keyed off a secret wrapped in a
+**SYSTEM-level** DPAPI layer plus app validation. `browser-cookie3` only
+knows the legacy `v10`/DPAPI key, so its AES-GCM MAC check fails — the exact
+symptom hit in the field:
 
-- Brave on Linux follows the Chromium pattern: cookies SQLite at
-  `~/.config/BraveSoftware/Brave-Browser/Default/Cookies`, values
-  AES-128-CBC encrypted with a key derived from the OS keyring secret
-  ("Brave Safe Storage" via SecretService/kwallet, or the `peanuts` v10
-  fallback). Well-trodden territory — evaluate a small dependency
-  (`browser_cookie3`-style) vs. a focused in-repo implementation.
-- Extract `__Secure-next-auth.session-token.0` / `.1` from `chatgpt.com`
-  and `sessionKey` from `claude.ai`; write to `.env` through the existing
-  auth-wizard write path (tokens never echoed).
-- Brave must be the user's logged-in browser (it is); detect a locked /
-  unreadable cookie DB (Brave running with the DB exclusively locked) and
-  fall back to the manual flow with a clear message.
-- Does **not** solve headless/StartOS — no browser on the server. That
-  needs a token-push mechanism, tracked under the StartOS entry.
+```
+ChatGPT: Could not read brave cookies for chatgpt.com: Unable to get key for cookie decryption.
+```
+
+Decrypting `v20` at all requires unwrapping the SYSTEM layer, which means
+running as SYSTEM (e.g. a PsExec-style service) — i.e. **Administrator
+rights** and behavior that AV/EDR flags as infostealer activity. The one
+maintained Python option (`rookiepy`) needs admin from Chrome v130+, was
+**archived 2026-06-07**, and has an unresolved bug where **Brave returns 0
+cookies** even after the key is retrieved. ABE is *designed* to stop exactly
+this, so no off-disk reader is a reliable, non-invasive fit.
+
+**If ever revisited:** the only non-admin path is Chrome Remote Debugging
+(launch the browser with `--remote-debugging-port`, read cookies via
+`Network.getAllCookies` — the running browser decrypts for you). Heavier and
+intrusive; not worth it for a weekly token refresh that takes 30 seconds by
+hand. With the headless/StartOS direction dropped (#8), manual DevTools
+refresh is the accepted approach — no automated extraction is needed.
 
 ## 3. Claude Code Session Provider — SHIPPED v0.6.0
 
@@ -257,43 +273,174 @@ Note: the conversation *listing* (paginated, 100/page) still runs in full
 each time so the cache comparison works — the cap applies to the heavy
 per-conversation detail fetches, which dominate request volume.
 
-Together with watch mode, this is a stepping stone to the StartOS service:
-a scheduled, capped, politely-paced export is the traffic profile a
-headless deployment needs.
+This is a stepping stone to the StartOS service: a capped, politely-paced
+export — scheduled by the host (cron/StartOS), not an in-app loop — is the
+traffic profile a headless deployment needs.
 
-## 7. Scheduled / Watch Mode
+## 7. Scheduled / Watch Mode — DROPPED (2026-06-28)
 
-Add a `watch` command (or cron integration helper) to run exports automatically
-on a schedule:
+An in-app `watch`/scheduler loop is not worth building. Scheduling belongs to
+whatever hosts the tool: a user cron line locally, and on the long-term
+StartOS target the platform's own scheduling. Either way the tool only needs
+to do one capped, politely-paced `export` + `joplin` run and exit — which it
+already does. If cron ergonomics ever feel clunky, a thin `sync` subcommand
+that chains `export` then `joplin` for a single cron line is a trivial
+add-on, but the polling loop itself is off the roadmap.
 
-```bash
-python -m src.main watch --interval 6h   # poll every 6 hours
-```
+## 8. StartOS Service Packaging — DROPPED (2026-06-28)
 
-This would run `export` + `joplin` in sequence, then sleep. Alternatively,
-provide a `cron` command that prints the correct crontab line for the user's
-setup.
+Not pursuing a headless StartOS service. The local, manually-run CLI is
+sufficient: the source conversations live in the providers' clouds and can be
+re-downloaded, and Joplin already syncs (encrypted) to an offsite S3 provider,
+so durability is covered without a server in the loop.
 
-Implementation: simple loop with `time.sleep()`, or emit a crontab entry
-string that calls the export and joplin commands in sequence. A `--once`
-flag would do a single run then exit (useful for cron itself).
+Dropping this also retires the one genuinely hard sub-problem it carried —
+session-token freshness without a browser. There is no headless context to
+keep fresh; the weekly manual DevTools refresh is acceptable. (Local cookie
+extraction remains a dead end regardless — see #2.)
 
-Stepping stone to the StartOS service (below).
+## 9. Token Validity on `doctor` — IMPLEMENTED (2026-06-28)
 
-## 8. StartOS Service Packaging (long-term destination)
+Shipped: `doctor` now adds a "ChatGPT token active" check via
+`ChatGPTProvider.session_health()` (reads `/api/auth/session`, passes iff
+`error` is falsy and `accessToken` is present), the never-working JWE/`exp`
+decode path was removed, and `_fetch_access_token` now fails fast on a set
+`error` instead of returning a stale token. Tests in
+`tests/test_providers.py::TestChatGPTSessionHealth`. Investigation trail
+below for the record.
 
-Run the exporter headless as a StartOS service: scheduled export + sync to
-a Joplin Server instance, so the backup happens without a desktop in the
-loop.
+Goal: if it's cheap to tell how much longer a token will work, show it on
+`doctor`. Findings from live recon:
 
-- Building blocks needed first: watch mode (#7), session limiter (#6),
-  and a Joplin Server (vs. desktop Web Clipper) sync target.
-- **Hard problem, flagged early:** session-token freshness without a
-  browser. ChatGPT tokens last ~7 days; there is no browser on the server
-  to re-extract from. Candidate mechanisms: a service config action the
-  user pastes fresh tokens into periodically, or a future browser
-  extension that pushes tokens to the server. Cookie auto-extraction (#2)
-  only solves the desktop case.
+- **Not readable from the token itself.** ChatGPT's `CHATGPT_SESSION_TOKEN`
+  is a **JWE** (header `{"alg":"dir","enc":"A256GCM"}`, `eyJ…` prefix is just
+  the encrypted protected header) — the `exp` claim is AES-256-GCM encrypted
+  with an OpenAI-only key, so it cannot be decoded client-side. Claude's
+  `sk-…` key is fully opaque. The existing `doctor` JWT-decode path therefore
+  never yields an expiry for the real tokens (falls to the "not decodable"
+  branch).
+- **`/api/auth/session` exposes an `expires`** (the provider already calls
+  this endpoint in `_fetch_access_token`; the response includes `expires`
+  alongside `accessToken`). Live value observed 2026-06-28:
+  `2026-09-26` — **~90 days out**. This contradicts both the code's ~7-day
+  assumption and the lived weekly-refresh cadence, so it is almost certainly
+  the NextAuth **rolling session window** (re-extended on every call), not
+  the point at which the pasted token actually 401s. Displaying it verbatim
+  would give false confidence.
+- **RESOLVED 2026-06-28 by a live 401 data point.** When the ChatGPT token
+  was actually dead (conversations API → 401), `/api/auth/session` still
+  returned **HTTP 200** with `expires: 2026-09-26` (~90 days out) — and that
+  `expires` *advanced* between two calls seconds apart (`04:20:08` → `04:30:37`).
+  So `expires` is a **rolling session window that rolls forward on every call
+  even for a dead token**; displaying it would actively lie. The same response
+  carried `error: "RefreshAccessTokenError"` and a stale `accessToken`.
+- **The real signal is `error`, not `expires` or `accessToken`.** On a healthy
+  token `error` is absent/null; when the session token is dead NextAuth can't
+  refresh and sets `error: "RefreshAccessTokenError"` while still echoing a
+  rolling `expires` and a stale `accessToken`. This is an exact, free, binary
+  health check.
+- **Design (ready to build):**
+  1. `doctor` ChatGPT check → read `/api/auth/session`; pass iff `error` is
+     falsy and `accessToken` present; on `RefreshAccessTokenError` report
+     "token expired — refresh". Drop the JWE/`exp` decode path (it can never
+     work) and do NOT surface `expires`.
+  2. Latent bug to fix alongside: `_fetch_access_token` reads `accessToken`
+     without checking `error`, so it proceeds with a stale token and yields a
+     confusing downstream 401 instead of a clear "refresh your token" message.
+     Check `error` there and fail fast.
+  3. Claude stays a 401-only signal (opaque `sk-`, no equivalent endpoint).
+
+## 10. Provider API-Drift Detection — IMPLEMENTED (2026-06-28)
+
+Shipped: the `canary` command + `BaseProvider.check_drift()` (overridden by
+ChatGPT and Claude). It fetches one listing page + one conversation per
+provider and asserts only the normalizer's load-bearing fields, emitting
+`DRIFT_OK/WARN/ERROR` findings (`src/providers/base.py`). Severity badges
+print as a Rich table; ERROR exits non-zero, WARN is non-fatal so a backup
+run is never blocked. Drift vocabularies (`_KNOWN_TOOL_AUTHORS`,
+`_HANDLED_CONTENT_TYPES`) live in `chatgpt.py` next to the collapse set they
+guard. Tests: `TestChatGPTDriftCanary`, `TestClaudeDriftCanary`,
+`TestCanaryCommand`. Verified live 2026-06-28 — both providers OK. Recon
+trail below for the record.
+
+## 10b. Provider API-Drift Detection — investigation (2026-06-28)
+
+The export depends on undocumented internal web APIs (ChatGPT/Claude) that can
+change shape without notice. The worst failure for a backup tool is *silent*:
+a response-schema change that makes the exporter skip or mis-parse content
+without erroring. `doctor` currently checks token validity, reachability, and
+manifest↔disk integrity — but not "does the provider's response still look
+like what the parser expects."
+
+To investigate: a lightweight schema/shape assertion on a known-good sample
+of each provider's listing + conversation-detail responses (presence and type
+of the fields the normalizers rely on), surfaced as a `doctor` check or a
+dedicated canary.
+
+**Live recon — Claude captured 2026-06-28 (ChatGPT pending a token refresh):**
+
+- **Dependency surface (assert ONLY these — see below for why):**
+  - listing item: `uuid`, `name`, `updated_at`/`created_at`, `project.name`.
+  - conversation detail: `uuid`/`id`, `name`, `created_at`, `updated_at`,
+    `project.name`, `chat_messages[]`.
+  - message: `sender` (`human`/`assistant`), `text` (string) or `content`
+    (list of typed blocks), `created_at`.
+- **Key finding — full-shape diffing is the wrong design.** Claude's
+  `settings` object is full of volatile internal codenames that churn
+  constantly: `enabled_bananagrams`, `enabled_sourdough`, `enabled_foccacia`,
+  `enabled_saffron`, `enabled_turmeric`, `enabled_monkeys_in_a_barrel`,
+  `paprika_mode`, `enabled_megaminds`, … A "any new/removed key = drift"
+  canary would fire on every UI experiment. The canary MUST target the
+  normalizer's load-bearing fields only, not the whole response. (Aligns with
+  the drop-noise-don't-retain-it principle.)
+- **Real Claude messages are flat `text`/`sender`** — in this archive every
+  message had a string `text` and NO `content` block list (0 rich blocks
+  observed). So `_extract_claude_blocks` / `_dispatch_claude_block` (tool_use,
+  thinking, image, …) is an **unexercised theoretical path**; drift there
+  can't be "caught" by a canary because it never runs on real data — it's a
+  safety net for if Claude ever switches to block content. The canary should
+  assert the flat shape and *warn if `content` ever appears as a list* (that
+  itself is the drift event that would activate the dormant code).
+- **Possible silent-loss spot (separate from drift):** Claude messages carry
+  `attachments` and `files` arrays (empty in this sample) that the normalizer
+  ignores entirely. If a user ever attaches files in Claude, they'd be
+  dropped without a LossReport entry. Worth a follow-up check.
+**Live recon — ChatGPT captured 2026-06-28:**
+
+- **Dependency surface (assert ONLY these):**
+  - listing item: `id`, `title`, `update_time`/`create_time`.
+  - conversation detail: `conversation_id`/`id`, `title`, `create_time`,
+    `update_time`, `mapping` (non-empty).
+  - mapping node: `message`, `children` (the tree walk depends on both);
+    message: `author.role`, `author.name`, `content.content_type`,
+    `content.parts`, `metadata.is_visually_hidden_from_conversation`.
+- **content_type vocabulary observed (all currently handled):** `text`,
+  `model_editable_context`, `multimodal_text`, `thoughts`, `code`,
+  `execution_output`, `reasoning_recap`, `user_editable_context`,
+  `tether_browsing_display`. A *new* content_type already degrades gracefully
+  (visible `unknown` block + WARNING + LossReport tally) — so content_type
+  drift is **already non-silent**. The canary just needs to confirm the known
+  set still parses to non-empty blocks.
+- **The genuinely silent drift risk — `author.name` collapse keys.**
+  `_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}`. Recon
+  confirms `file_search` is live (and `web.run`/`python` are correctly left
+  un-collapsed). If OpenAI renames `file_search`, the collapse **silently
+  stops** and the archive re-bloats with no error or LossReport entry. This is
+  the top canary target: assert that retrieval-dump tool authors are still
+  recognized, or at least flag unfamiliar `(role="tool", author.name)` pairs.
+- **Second silent risk — empty `content.parts`.** A `text` message whose
+  `parts` field is renamed/emptied yields zero blocks and is skipped with only
+  a debug log = silent loss. Canary should assert a sampled `text` message
+  produces a non-empty block.
+
+**Recon complete for both providers. Canary design (ready to build):** a
+`doctor` check (or dedicated `canary` command) that, per provider, fetches one
+listing page + one conversation and asserts the dependency-surface fields
+above by presence+type — NOT full shape (Claude `settings` codenames prove
+full-shape diffing is pure noise). Specific tripwires: (ChatGPT) unfamiliar
+`(tool, author.name)` pair and empty `parts` on a text message; (Claude)
+`content` appearing as a list, and non-empty `attachments`/`files`. Failures
+surface as a warning, never a hard error (a backup tool must still run).
 
 ---
 
@@ -369,16 +516,13 @@ structure matching the user's Obsidian setup. No API needed — just file I/O.
 
 ## Token Expiry Notifications
 
-Proactively warn when a token is close to expiry (within 48h for ChatGPT),
-rather than only surfacing the warning at startup. Options:
-
-- Add an `expiry` subcommand that prints token status and exits non-zero if
-  any token is expired or expiring soon (useful in scripts/cron)
-- Send a desktop notification via `notify-send` (Linux) or `osascript` (macOS)
-  when a token is within 24h of expiry
-
-Largely superseded by Brave cookie auto-extraction (#2): once refresh is
-one command (or automatic), expiry warnings matter much less.
+Moved to the active roadmap as §9 (Token Validity on `doctor`). The original
+"proactively notify before expiry" idea is blocked by the same finding: a
+reliable expiry time isn't available client-side (ChatGPT token is encrypted,
+Claude's is opaque, and `/api/auth/session`'s `expires` looks like a rolling
+window rather than the real refresh cadence). Any heads-up — a `doctor`
+line, an `expiry` subcommand, or a `notify-send` nudge — depends on first
+resolving the §9 open question of what signal is actually trustworthy.
 
 ## Search Command