canary: detect provider API schema drift; release v0.7.0
The export reads ChatGPT's and Claude's undocumented internal web APIs, which can change shape without notice; the worst failure for a backup tool is a silent one (skipped/mis-parsed content with no error). Add a `canary` command + BaseProvider.check_drift() (overridden by ChatGPT/Claude) that fetches one listing page + one conversation per provider and asserts only the normalizer's load-bearing fields — not the full response shape, which churns harmlessly. The top silent risk it guards is a renamed retrieval-tool author bypassing the hidden-content collapse. ERROR findings exit non-zero; WARN findings are surfaced but non-fatal so a backup run is never blocked. Also retires the in-app watch mode and headless StartOS direction from the roadmap (tool stays a local, manually-run CLI) and updates docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
ef603cf659
commit
1e016ea652
@@ -3,6 +3,18 @@
|
|||||||
All notable changes to this project will be documented here.
|
All notable changes to this project will be documented here.
|
||||||
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||||
|
|
||||||
|
## [0.7.0] - 2026-06-28
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- `canary` command — probes the ChatGPT/Claude web APIs for schema drift against the fields the parser actually depends on (one listing page + one conversation per provider), not the full response shape. Flags the silent-failure risks for a backup tool: a renamed retrieval-tool author bypassing the hidden-content collapse, a new `content_type`, drifted message fields, or non-empty Claude `attachments`/`files` the normalizer ignores. ERROR findings exit non-zero; WARN findings are surfaced but non-fatal. See `FUTURE.md` §10.
|
||||||
|
- `doctor` now reports a real ChatGPT token-health check ("ChatGPT token active") based on the `/api/auth/session` `error` field. ChatGPT session tokens are JWEs whose `exp` is encrypted and unreadable client-side; the previous decode path could never yield an expiry. The honest signal is `error == "RefreshAccessTokenError"` (verified live), which means the session token is dead even though `expires`/`accessToken` still echo stale values. See `FUTURE.md` §9.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- `ChatGPTProvider._fetch_access_token` now checks the `/api/auth/session` `error` field and fails fast with a clear "refresh your token" message. Previously it returned the stale `accessToken` present on a dead session, producing a confusing downstream 401 instead of an actionable auth error.
|
||||||
|
|
||||||
|
### Removed
|
||||||
|
- `auth --from-browser` and the `browser-cookie3` dependency (`src/browser_tokens.py`). Browser cookie auto-extraction is not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave) keys cookies off a SYSTEM-level layer that can't be decrypted off disk without Administrator rights and AV-flagged SYSTEM impersonation, and fails on Brave specifically (symptom: `Unable to get key for cookie decryption`). Auth is manual (DevTools wizard) only. See `FUTURE.md` §2 for the full rationale.
|
||||||
|
|
||||||
## [0.6.0] - 2026-06-12
|
## [0.6.0] - 2026-06-12
|
||||||
|
|
||||||
First tagged release. The project was developed through several internal
|
First tagged release. The project was developed through several internal
|
||||||
|
|||||||
@@ -17,14 +17,18 @@ of these additions straightforward.
|
|||||||
|
|
||||||
Priorities reflect the tool's primary purpose — a trustworthy backup so that
|
Priorities reflect the tool's primary purpose — a trustworthy backup so that
|
||||||
conversation data is not lost if a provider account is ever closed — plus the
|
conversation data is not lost if a provider account is ever closed — plus the
|
||||||
day-to-day friction of weekly ChatGPT token refresh, and the long-term goal
|
day-to-day friction of the weekly ChatGPT token refresh. The tool stays a
|
||||||
of running headless as a StartOS service.
|
local, manually-run CLI; the headless/StartOS direction was dropped
|
||||||
|
2026-06-28 (see #7 and #8), which also retires the token-freshness problem
|
||||||
|
(manual refresh is sufficient).
|
||||||
|
|
||||||
**Now (in order):**
|
**Now (in order):**
|
||||||
1. ~~Collapse tool retrieval dumps & hidden context~~ — **shipped in v0.6.0**
|
1. ~~Collapse tool retrieval dumps & hidden context~~ — **shipped in v0.6.0**
|
||||||
(full-archive re-export still pending; see entry below)
|
(full-archive `export --force` re-export completed 2026-06-13)
|
||||||
2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0** (note: runs on
|
2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0, removed afterward.**
|
||||||
whichever machine hosts the browser — Brave is not on this Linux box)
|
Not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave)
|
||||||
|
needs admin + SYSTEM impersonation that AV flags as credential theft, and
|
||||||
|
fails on Brave specifically. Auth is manual (DevTools) — see entry below.
|
||||||
3. ~~Claude Code session provider~~ — **shipped in v0.6.0**
|
3. ~~Claude Code session provider~~ — **shipped in v0.6.0**
|
||||||
4. ~~Archive hygiene — `prune` + `doctor` integrity check~~ — **shipped in v0.6.0**
|
4. ~~Archive hygiene — `prune` + `doctor` integrity check~~ — **shipped in v0.6.0**
|
||||||
|
|
||||||
@@ -32,12 +36,23 @@ of running headless as a StartOS service.
|
|||||||
|
|
||||||
5. ~~Binary content downloads~~ — **shipped in v0.6.0**
|
5. ~~Binary content downloads~~ — **shipped in v0.6.0**
|
||||||
6. ~~Per-session download limiter + polite pacing~~ — **shipped in v0.6.0**
|
6. ~~Per-session download limiter + polite pacing~~ — **shipped in v0.6.0**
|
||||||
7. Scheduled / watch mode → StartOS service packaging (long-term destination)
|
7. ~~Scheduled / watch mode~~ — **dropped 2026-06-28**; the tool stays a
|
||||||
|
manually-run CLI, so no in-app polling loop is needed
|
||||||
|
8. ~~StartOS service packaging~~ — **dropped 2026-06-28**; the local CLI is
|
||||||
|
sufficient (source convos live in the cloud and can be re-downloaded;
|
||||||
|
Joplin already syncs encrypted to an offsite S3 provider). Dropping this
|
||||||
|
also retires the headless token-freshness problem — manual weekly refresh
|
||||||
|
is fine.
|
||||||
|
|
||||||
|
**Active:**
|
||||||
|
9. ~~Surface remaining token validity on `doctor`~~ — **IMPLEMENTED 2026-06-28**
|
||||||
|
via the `/api/auth/session` `error` field (not `expires`). See §9.
|
||||||
|
10. ~~Provider API-drift detection~~ — **IMPLEMENTED 2026-06-28** as the
|
||||||
|
`canary` command. See §10.
|
||||||
|
|
||||||
**Deprioritized** (entries kept at the bottom of this file; revisit on
|
**Deprioritized** (entries kept at the bottom of this file; revisit on
|
||||||
demand): `--force` flags, per-conversation cache reset, official export-ZIP
|
demand): `--force` flags, per-conversation cache reset, official export-ZIP
|
||||||
fallback, o1/o3 reasoning reclassification, Obsidian output, token expiry
|
fallback, o1/o3 reasoning reclassification, Obsidian output, search command.
|
||||||
notifications (largely superseded by cookie auto-extraction), search command.
|
|
||||||
Additional web providers (Gemini/Grok/Perplexity) are explicitly out of
|
Additional web providers (Gemini/Grok/Perplexity) are explicitly out of
|
||||||
scope — no significant usage to archive.
|
scope — no significant usage to archive.
|
||||||
|
|
||||||
@@ -50,8 +65,8 @@ from live recon: the retrieval dumps are NOT flagged
|
|||||||
`is_visually_hidden_from_conversation` — `author.name == "file_search"` is
|
`is_visually_hidden_from_conversation` — `author.name == "file_search"` is
|
||||||
the discriminator (the hidden flag only marks Custom Instructions and small
|
the discriminator (the hidden flag only marks Custom Instructions and small
|
||||||
system stubs). Verified live: worst files shrink 93% (524KB → 36KB).
|
system stubs). Verified live: worst files shrink 93% (524KB → 36KB).
|
||||||
Remaining step: full-archive re-export (`cache --clear` + `export`) +
|
Full-archive re-export (the `export --force` campaign) + Joplin re-sync
|
||||||
Joplin re-sync, at the user's chosen time.
|
completed 2026-06-13.
|
||||||
|
|
||||||
**Problem (measured 2026-06-12 against a full fresh export):** 45% of the
|
**Problem (measured 2026-06-12 against a full fresh export):** 45% of the
|
||||||
entire 11.2MB archive (260 files) is tool-role messages; 29 files are
|
entire 11.2MB archive (260 files) is tool-role messages; 29 files are
|
||||||
@@ -89,37 +104,38 @@ file contains zero hidden-context blocks.
|
|||||||
Re-export workflow after shipping: `cache --clear` + `export` (same as
|
Re-export workflow after shipping: `cache --clear` + `export` (same as
|
||||||
the v0.4.0 migration).
|
the v0.4.0 migration).
|
||||||
|
|
||||||
## 2. Brave Cookie Auto-Extraction — SHIPPED v0.6.0
|
## 2. Brave Cookie Auto-Extraction — REMOVED (not viable)
|
||||||
|
|
||||||
**Implemented 2026-06-12**: `auth --from-browser [browser]`
|
**Shipped v0.6.0 (2026-06-12), removed 2026-06-27.** `auth --from-browser`
|
||||||
(`src/browser_tokens.py`, browser-cookie3 dependency, defaults to brave;
|
plus `src/browser_tokens.py` and the `browser-cookie3` dependency are gone.
|
||||||
chrome/chromium/edge/firefox also supported). Extracted tokens are validated
|
Auth is manual (DevTools) only. Do not re-attempt without a fundamentally
|
||||||
against the live API before `.env` is touched. **Discovered during
|
different mechanism (see below).
|
||||||
implementation: Brave is not installed on this machine** — the logged-in
|
|
||||||
browser lives elsewhere, so the flag only helps when the exporter runs on
|
|
||||||
that machine (verified the graceful-failure path here; the happy path is
|
|
||||||
covered by mocked tests). This strengthens the case for the StartOS
|
|
||||||
token-push mechanism (entry 8).
|
|
||||||
|
|
||||||
Pull session tokens directly from the Brave browser profile instead of the
|
**Why it doesn't work.** Modern Chromium browsers encrypt cookies on Windows
|
||||||
weekly manual DevTools dance. `auth` gains an "extract from browser" path
|
with **App-Bound Encryption** (Chrome 127+, July 2024; current Brave).
|
||||||
(with the manual flow kept as fallback); a later iteration could let
|
Cookies are written with a `v20` prefix and keyed off a secret wrapped in a
|
||||||
`doctor` or a 401 handler suggest/perform a re-extract automatically.
|
**SYSTEM-level** DPAPI layer plus app validation. `browser-cookie3` only
|
||||||
|
knows the legacy `v10`/DPAPI key, so its AES-GCM MAC check fails — the exact
|
||||||
|
symptom hit in the field:
|
||||||
|
|
||||||
- Brave on Linux follows the Chromium pattern: cookies SQLite at
|
```
|
||||||
`~/.config/BraveSoftware/Brave-Browser/Default/Cookies`, values
|
ChatGPT: Could not read brave cookies for chatgpt.com: Unable to get key for cookie decryption.
|
||||||
AES-128-CBC encrypted with a key derived from the OS keyring secret
|
```
|
||||||
("Brave Safe Storage" via SecretService/kwallet, or the `peanuts` v10
|
|
||||||
fallback). Well-trodden territory — evaluate a small dependency
|
Decrypting `v20` at all requires unwrapping the SYSTEM layer, which means
|
||||||
(`browser_cookie3`-style) vs. a focused in-repo implementation.
|
running as SYSTEM (e.g. a PsExec-style service) — i.e. **Administrator
|
||||||
- Extract `__Secure-next-auth.session-token.0` / `.1` from `chatgpt.com`
|
rights** and behavior that AV/EDR flags as infostealer activity. The one
|
||||||
and `sessionKey` from `claude.ai`; write to `.env` through the existing
|
maintained Python option (`rookiepy`) needs admin from Chrome v130+, was
|
||||||
auth-wizard write path (tokens never echoed).
|
**archived 2026-06-07**, and has an unresolved bug where **Brave returns 0
|
||||||
- Brave must be the user's logged-in browser (it is); detect a locked /
|
cookies** even after the key is retrieved. ABE is *designed* to stop exactly
|
||||||
unreadable cookie DB (Brave running with the DB exclusively locked) and
|
this, so no off-disk reader is a reliable, non-invasive fit.
|
||||||
fall back to the manual flow with a clear message.
|
|
||||||
- Does **not** solve headless/StartOS — no browser on the server. That
|
**If ever revisited:** the only non-admin path is Chrome Remote Debugging
|
||||||
needs a token-push mechanism, tracked under the StartOS entry.
|
(launch the browser with `--remote-debugging-port`, read cookies via
|
||||||
|
`Network.getAllCookies` — the running browser decrypts for you). Heavier and
|
||||||
|
intrusive; not worth it for a weekly token refresh that takes 30 seconds by
|
||||||
|
hand. With the headless/StartOS direction dropped (#8), manual DevTools
|
||||||
|
refresh is the accepted approach — no automated extraction is needed.
|
||||||
|
|
||||||
## 3. Claude Code Session Provider — SHIPPED v0.6.0
|
## 3. Claude Code Session Provider — SHIPPED v0.6.0
|
||||||
|
|
||||||
@@ -257,43 +273,174 @@ Note: the conversation *listing* (paginated, 100/page) still runs in full
|
|||||||
each time so the cache comparison works — the cap applies to the heavy
|
each time so the cache comparison works — the cap applies to the heavy
|
||||||
per-conversation detail fetches, which dominate request volume.
|
per-conversation detail fetches, which dominate request volume.
|
||||||
|
|
||||||
Together with watch mode, this is a stepping stone to the StartOS service:
|
This is a stepping stone to the StartOS service: a capped, politely-paced
|
||||||
a scheduled, capped, politely-paced export is the traffic profile a
|
export — scheduled by the host (cron/StartOS), not an in-app loop — is the
|
||||||
headless deployment needs.
|
traffic profile a headless deployment needs.
|
||||||
|
|
||||||
## 7. Scheduled / Watch Mode
|
## 7. Scheduled / Watch Mode — DROPPED (2026-06-28)
|
||||||
|
|
||||||
Add a `watch` command (or cron integration helper) to run exports automatically
|
An in-app `watch`/scheduler loop is not worth building. Scheduling belongs to
|
||||||
on a schedule:
|
whatever hosts the tool: a user cron line locally, and on the long-term
|
||||||
|
StartOS target the platform's own scheduling. Either way the tool only needs
|
||||||
|
to do one capped, politely-paced `export` + `joplin` run and exit — which it
|
||||||
|
already does. If cron ergonomics ever feel clunky, a thin `sync` subcommand
|
||||||
|
that chains `export` then `joplin` for a single cron line is a trivial
|
||||||
|
add-on, but the polling loop itself is off the roadmap.
|
||||||
|
|
||||||
```bash
|
## 8. StartOS Service Packaging — DROPPED (2026-06-28)
|
||||||
python -m src.main watch --interval 6h # poll every 6 hours
|
|
||||||
```
|
|
||||||
|
|
||||||
This would run `export` + `joplin` in sequence, then sleep. Alternatively,
|
Not pursuing a headless StartOS service. The local, manually-run CLI is
|
||||||
provide a `cron` command that prints the correct crontab line for the user's
|
sufficient: the source conversations live in the providers' clouds and can be
|
||||||
setup.
|
re-downloaded, and Joplin already syncs (encrypted) to an offsite S3 provider,
|
||||||
|
so durability is covered without a server in the loop.
|
||||||
|
|
||||||
Implementation: simple loop with `time.sleep()`, or emit a crontab entry
|
Dropping this also retires the one genuinely hard sub-problem it carried —
|
||||||
string that calls the export and joplin commands in sequence. A `--once`
|
session-token freshness without a browser. There is no headless context to
|
||||||
flag would do a single run then exit (useful for cron itself).
|
keep fresh; the weekly manual DevTools refresh is acceptable. (Local cookie
|
||||||
|
extraction remains a dead end regardless — see #2.)
|
||||||
|
|
||||||
Stepping stone to the StartOS service (below).
|
## 9. Token Validity on `doctor` — IMPLEMENTED (2026-06-28)
|
||||||
|
|
||||||
## 8. StartOS Service Packaging (long-term destination)
|
Shipped: `doctor` now adds a "ChatGPT token active" check via
|
||||||
|
`ChatGPTProvider.session_health()` (reads `/api/auth/session`, passes iff
|
||||||
|
`error` is falsy and `accessToken` is present), the never-working JWE/`exp`
|
||||||
|
decode path was removed, and `_fetch_access_token` now fails fast on a set
|
||||||
|
`error` instead of returning a stale token. Tests in
|
||||||
|
`tests/test_providers.py::TestChatGPTSessionHealth`. Investigation trail
|
||||||
|
below for the record.
|
||||||
|
|
||||||
Run the exporter headless as a StartOS service: scheduled export + sync to
|
Goal: if it's cheap to tell how much longer a token will work, show it on
|
||||||
a Joplin Server instance, so the backup happens without a desktop in the
|
`doctor`. Findings from live recon:
|
||||||
loop.
|
|
||||||
|
|
||||||
- Building blocks needed first: watch mode (#7), session limiter (#6),
|
- **Not readable from the token itself.** ChatGPT's `CHATGPT_SESSION_TOKEN`
|
||||||
and a Joplin Server (vs. desktop Web Clipper) sync target.
|
is a **JWE** (header `{"alg":"dir","enc":"A256GCM"}`, `eyJ…` prefix is just
|
||||||
- **Hard problem, flagged early:** session-token freshness without a
|
the encrypted protected header) — the `exp` claim is AES-256-GCM encrypted
|
||||||
browser. ChatGPT tokens last ~7 days; there is no browser on the server
|
with an OpenAI-only key, so it cannot be decoded client-side. Claude's
|
||||||
to re-extract from. Candidate mechanisms: a service config action the
|
`sk-…` key is fully opaque. The existing `doctor` JWT-decode path therefore
|
||||||
user pastes fresh tokens into periodically, or a future browser
|
never yields an expiry for the real tokens (falls to the "not decodable"
|
||||||
extension that pushes tokens to the server. Cookie auto-extraction (#2)
|
branch).
|
||||||
only solves the desktop case.
|
- **`/api/auth/session` exposes an `expires`** (the provider already calls
|
||||||
|
this endpoint in `_fetch_access_token`; the response includes `expires`
|
||||||
|
alongside `accessToken`). Live value observed 2026-06-28:
|
||||||
|
`2026-09-26` — **~90 days out**. This contradicts both the code's ~7-day
|
||||||
|
assumption and the lived weekly-refresh cadence, so it is almost certainly
|
||||||
|
the NextAuth **rolling session window** (re-extended on every call), not
|
||||||
|
the point at which the pasted token actually 401s. Displaying it verbatim
|
||||||
|
would give false confidence.
|
||||||
|
- **RESOLVED 2026-06-28 by a live 401 data point.** When the ChatGPT token
|
||||||
|
was actually dead (conversations API → 401), `/api/auth/session` still
|
||||||
|
returned **HTTP 200** with `expires: 2026-09-26` (~90 days out) — and that
|
||||||
|
`expires` *advanced* between two calls seconds apart (`04:20:08` → `04:30:37`).
|
||||||
|
So `expires` is a **rolling session window that rolls forward on every call
|
||||||
|
even for a dead token**; displaying it would actively lie. The same response
|
||||||
|
carried `error: "RefreshAccessTokenError"` and a stale `accessToken`.
|
||||||
|
- **The real signal is `error`, not `expires` or `accessToken`.** On a healthy
|
||||||
|
token `error` is absent/null; when the session token is dead NextAuth can't
|
||||||
|
refresh and sets `error: "RefreshAccessTokenError"` while still echoing a
|
||||||
|
rolling `expires` and a stale `accessToken`. This is an exact, free, binary
|
||||||
|
health check.
|
||||||
|
- **Design (ready to build):**
|
||||||
|
1. `doctor` ChatGPT check → read `/api/auth/session`; pass iff `error` is
|
||||||
|
falsy and `accessToken` present; on `RefreshAccessTokenError` report
|
||||||
|
"token expired — refresh". Drop the JWE/`exp` decode path (it can never
|
||||||
|
work) and do NOT surface `expires`.
|
||||||
|
2. Latent bug to fix alongside: `_fetch_access_token` reads `accessToken`
|
||||||
|
without checking `error`, so it proceeds with a stale token and yields a
|
||||||
|
confusing downstream 401 instead of a clear "refresh your token" message.
|
||||||
|
Check `error` there and fail fast.
|
||||||
|
3. Claude stays a 401-only signal (opaque `sk-`, no equivalent endpoint).
|
||||||
|
|
||||||
|
## 10. Provider API-Drift Detection — IMPLEMENTED (2026-06-28)
|
||||||
|
|
||||||
|
Shipped: the `canary` command + `BaseProvider.check_drift()` (overridden by
|
||||||
|
ChatGPT and Claude). It fetches one listing page + one conversation per
|
||||||
|
provider and asserts only the normalizer's load-bearing fields, emitting
|
||||||
|
`DRIFT_OK/WARN/ERROR` findings (`src/providers/base.py`). Severity badges
|
||||||
|
print as a Rich table; ERROR exits non-zero, WARN is non-fatal so a backup
|
||||||
|
run is never blocked. Drift vocabularies (`_KNOWN_TOOL_AUTHORS`,
|
||||||
|
`_HANDLED_CONTENT_TYPES`) live in `chatgpt.py` next to the collapse set they
|
||||||
|
guard. Tests: `TestChatGPTDriftCanary`, `TestClaudeDriftCanary`,
|
||||||
|
`TestCanaryCommand`. Verified live 2026-06-28 — both providers OK. Recon
|
||||||
|
trail below for the record.
|
||||||
|
|
||||||
|
## 10b. Provider API-Drift Detection — investigation (2026-06-28)
|
||||||
|
|
||||||
|
The export depends on undocumented internal web APIs (ChatGPT/Claude) that can
|
||||||
|
change shape without notice. The worst failure for a backup tool is *silent*:
|
||||||
|
a response-schema change that makes the exporter skip or mis-parse content
|
||||||
|
without erroring. `doctor` currently checks token validity, reachability, and
|
||||||
|
manifest↔disk integrity — but not "does the provider's response still look
|
||||||
|
like what the parser expects."
|
||||||
|
|
||||||
|
To investigate: a lightweight schema/shape assertion on a known-good sample
|
||||||
|
of each provider's listing + conversation-detail responses (presence and type
|
||||||
|
of the fields the normalizers rely on), surfaced as a `doctor` check or a
|
||||||
|
dedicated canary.
|
||||||
|
|
||||||
|
**Live recon — Claude captured 2026-06-28 (ChatGPT pending a token refresh):**
|
||||||
|
|
||||||
|
- **Dependency surface (assert ONLY these — see below for why):**
|
||||||
|
- listing item: `uuid`, `name`, `updated_at`/`created_at`, `project.name`.
|
||||||
|
- conversation detail: `uuid`/`id`, `name`, `created_at`, `updated_at`,
|
||||||
|
`project.name`, `chat_messages[]`.
|
||||||
|
- message: `sender` (`human`/`assistant`), `text` (string) or `content`
|
||||||
|
(list of typed blocks), `created_at`.
|
||||||
|
- **Key finding — full-shape diffing is the wrong design.** Claude's
|
||||||
|
`settings` object is full of volatile internal codenames that churn
|
||||||
|
constantly: `enabled_bananagrams`, `enabled_sourdough`, `enabled_foccacia`,
|
||||||
|
`enabled_saffron`, `enabled_turmeric`, `enabled_monkeys_in_a_barrel`,
|
||||||
|
`paprika_mode`, `enabled_megaminds`, … A "any new/removed key = drift"
|
||||||
|
canary would fire on every UI experiment. The canary MUST target the
|
||||||
|
normalizer's load-bearing fields only, not the whole response. (Aligns with
|
||||||
|
the drop-noise-don't-retain-it principle.)
|
||||||
|
- **Real Claude messages are flat `text`/`sender`** — in this archive every
|
||||||
|
message had a string `text` and NO `content` block list (0 rich blocks
|
||||||
|
observed). So `_extract_claude_blocks` / `_dispatch_claude_block` (tool_use,
|
||||||
|
thinking, image, …) is an **unexercised theoretical path**; drift there
|
||||||
|
can't be "caught" by a canary because it never runs on real data — it's a
|
||||||
|
safety net for if Claude ever switches to block content. The canary should
|
||||||
|
assert the flat shape and *warn if `content` ever appears as a list* (that
|
||||||
|
itself is the drift event that would activate the dormant code).
|
||||||
|
- **Possible silent-loss spot (separate from drift):** Claude messages carry
|
||||||
|
`attachments` and `files` arrays (empty in this sample) that the normalizer
|
||||||
|
ignores entirely. If a user ever attaches files in Claude, they'd be
|
||||||
|
dropped without a LossReport entry. Worth a follow-up check.
|
||||||
|
**Live recon — ChatGPT captured 2026-06-28:**
|
||||||
|
|
||||||
|
- **Dependency surface (assert ONLY these):**
|
||||||
|
- listing item: `id`, `title`, `update_time`/`create_time`.
|
||||||
|
- conversation detail: `conversation_id`/`id`, `title`, `create_time`,
|
||||||
|
`update_time`, `mapping` (non-empty).
|
||||||
|
- mapping node: `message`, `children` (the tree walk depends on both);
|
||||||
|
message: `author.role`, `author.name`, `content.content_type`,
|
||||||
|
`content.parts`, `metadata.is_visually_hidden_from_conversation`.
|
||||||
|
- **content_type vocabulary observed (all currently handled):** `text`,
|
||||||
|
`model_editable_context`, `multimodal_text`, `thoughts`, `code`,
|
||||||
|
`execution_output`, `reasoning_recap`, `user_editable_context`,
|
||||||
|
`tether_browsing_display`. A *new* content_type already degrades gracefully
|
||||||
|
(visible `unknown` block + WARNING + LossReport tally) — so content_type
|
||||||
|
drift is **already non-silent**. The canary just needs to confirm the known
|
||||||
|
set still parses to non-empty blocks.
|
||||||
|
- **The genuinely silent drift risk — `author.name` collapse keys.**
|
||||||
|
`_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}`. Recon
|
||||||
|
confirms `file_search` is live (and `web.run`/`python` are correctly left
|
||||||
|
un-collapsed). If OpenAI renames `file_search`, the collapse **silently
|
||||||
|
stops** and the archive re-bloats with no error or LossReport entry. This is
|
||||||
|
the top canary target: assert that retrieval-dump tool authors are still
|
||||||
|
recognized, or at least flag unfamiliar `(role="tool", author.name)` pairs.
|
||||||
|
- **Second silent risk — empty `content.parts`.** A `text` message whose
|
||||||
|
`parts` field is renamed/emptied yields zero blocks and is skipped with only
|
||||||
|
a debug log = silent loss. Canary should assert a sampled `text` message
|
||||||
|
produces a non-empty block.
|
||||||
|
|
||||||
|
**Recon complete for both providers. Canary design (ready to build):** a
|
||||||
|
`doctor` check (or dedicated `canary` command) that, per provider, fetches one
|
||||||
|
listing page + one conversation and asserts the dependency-surface fields
|
||||||
|
above by presence+type — NOT full shape (Claude `settings` codenames prove
|
||||||
|
full-shape diffing is pure noise). Specific tripwires: (ChatGPT) unfamiliar
|
||||||
|
`(tool, author.name)` pair and empty `parts` on a text message; (Claude)
|
||||||
|
`content` appearing as a list, and non-empty `attachments`/`files`. Failures
|
||||||
|
surface as a warning, never a hard error (a backup tool must still run).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -369,16 +516,13 @@ structure matching the user's Obsidian setup. No API needed — just file I/O.
|
|||||||
|
|
||||||
## Token Expiry Notifications
|
## Token Expiry Notifications
|
||||||
|
|
||||||
Proactively warn when a token is close to expiry (within 48h for ChatGPT),
|
Moved to the active roadmap as §9 (Token Validity on `doctor`). The original
|
||||||
rather than only surfacing the warning at startup. Options:
|
"proactively notify before expiry" idea is blocked by the same finding: a
|
||||||
|
reliable expiry time isn't available client-side (ChatGPT token is encrypted,
|
||||||
- Add an `expiry` subcommand that prints token status and exits non-zero if
|
Claude's is opaque, and `/api/auth/session`'s `expires` looks like a rolling
|
||||||
any token is expired or expiring soon (useful in scripts/cron)
|
window rather than the real refresh cadence). Any heads-up — a `doctor`
|
||||||
- Send a desktop notification via `notify-send` (Linux) or `osascript` (macOS)
|
line, an `expiry` subcommand, or a `notify-send` nudge — depends on first
|
||||||
when a token is within 24h of expiry
|
resolving the §9 open question of what signal is actually trustworthy.
|
||||||
|
|
||||||
Largely superseded by Brave cookie auto-extraction (#2): once refresh is
|
|
||||||
one command (or automatic), expiry warnings matter much less.
|
|
||||||
|
|
||||||
## Search Command
|
## Search Command
|
||||||
|
|
||||||
|
|||||||
@@ -79,7 +79,17 @@ Before anything else, validate your setup:
|
|||||||
ai-chat-exporter doctor
|
ai-chat-exporter doctor
|
||||||
```
|
```
|
||||||
|
|
||||||
This checks token presence, format, expiry, directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.
|
This checks token presence, format, token health (via the `/api/auth/session` `error` field), directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.
|
||||||
|
|
||||||
|
### Checking for API drift
|
||||||
|
|
||||||
|
The exporter reads ChatGPT's and Claude's undocumented internal web APIs, which can change shape without notice. The worst failure for a backup tool is *silent* — a response change that makes the exporter skip or mis-parse content without erroring. Run the canary to catch that early:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ai-chat-exporter canary
|
||||||
|
```
|
||||||
|
|
||||||
|
It probes one conversation per provider and checks only the fields the parser depends on (a renamed retrieval-tool author that would bypass the content-collapse, a new content type, drifted message fields, ignored attachments). `OK` means the live schema still matches; `WARN`/`ERROR` tells you exactly what changed. Worth running before a large export or on a schedule.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "ai-chat-exporter"
|
name = "ai-chat-exporter"
|
||||||
version = "0.6.0"
|
version = "0.7.0"
|
||||||
description = "Export ChatGPT and Claude conversation history to Markdown for personal archival in Joplin"
|
description = "Export ChatGPT and Claude conversation history to Markdown for personal archival in Joplin"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
|
|||||||
+86
-1
@@ -18,7 +18,7 @@ from src.cache import Cache, CacheError
|
|||||||
from src.config import ConfigError
|
from src.config import ConfigError
|
||||||
from src.logging_config import setup_logging
|
from src.logging_config import setup_logging
|
||||||
from src.loss_report import LossReport
|
from src.loss_report import LossReport
|
||||||
from src.providers.base import ProviderError
|
from src.providers.base import DRIFT_ERROR, DRIFT_OK, DRIFT_WARN, ProviderError
|
||||||
|
|
||||||
console = Console()
|
console = Console()
|
||||||
err_console = Console(stderr=True)
|
err_console = Console(stderr=True)
|
||||||
@@ -332,6 +332,91 @@ def doctor(ctx: click.Context) -> None:
|
|||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
@cli.command()
|
||||||
|
@click.option(
|
||||||
|
"--provider",
|
||||||
|
type=click.Choice(["chatgpt", "claude", "all"]),
|
||||||
|
default="all",
|
||||||
|
help="Which web-API provider(s) to probe.",
|
||||||
|
)
|
||||||
|
@click.pass_context
|
||||||
|
def canary(ctx: click.Context, provider: str) -> None:
|
||||||
|
"""Probe provider APIs for schema drift against the fields the parser needs.
|
||||||
|
|
||||||
|
Fetches one listing page + one conversation per provider and checks only
|
||||||
|
the normalizer's load-bearing fields (not the full response shape, which
|
||||||
|
churns harmlessly). Surfaces the silent-failure risks for a backup tool:
|
||||||
|
a renamed retrieval-tool author bypassing the hidden-content collapse, a
|
||||||
|
new content_type, drifted message fields, or ignored attachments.
|
||||||
|
|
||||||
|
Exits non-zero if any ERROR finding is reported (a load-bearing field is
|
||||||
|
missing/mistyped). WARN findings are surfaced but do not fail. See
|
||||||
|
FUTURE.md §10.
|
||||||
|
"""
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
load_dotenv(override=False)
|
||||||
|
|
||||||
|
chatgpt_token = os.getenv("CHATGPT_SESSION_TOKEN", "").strip() or None
|
||||||
|
claude_key = os.getenv("CLAUDE_SESSION_KEY", "").strip() or None
|
||||||
|
|
||||||
|
targets: list[tuple[str, object]] = []
|
||||||
|
if provider in ("chatgpt", "all") and chatgpt_token:
|
||||||
|
from src.providers.chatgpt import ChatGPTProvider
|
||||||
|
|
||||||
|
ct1 = os.getenv("CHATGPT_SESSION_TOKEN_1", "").strip() or None
|
||||||
|
targets.append(
|
||||||
|
("chatgpt", lambda: ChatGPTProvider(chatgpt_token, session_token_1=ct1))
|
||||||
|
)
|
||||||
|
if provider in ("claude", "all") and claude_key:
|
||||||
|
from src.providers.claude import ClaudeProvider
|
||||||
|
|
||||||
|
targets.append(("claude", lambda: ClaudeProvider(claude_key)))
|
||||||
|
|
||||||
|
if not targets:
|
||||||
|
err_console.print(
|
||||||
|
"[yellow]No web-API provider tokens configured "
|
||||||
|
"(set CHATGPT_SESSION_TOKEN / CLAUDE_SESSION_KEY).[/yellow]"
|
||||||
|
)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
rows: list[tuple[str, dict]] = []
|
||||||
|
for name, factory in targets:
|
||||||
|
try:
|
||||||
|
findings = factory().check_drift()
|
||||||
|
except ProviderError as e:
|
||||||
|
findings = [{"severity": DRIFT_ERROR, "check": "probe", "detail": str(e.original)[:100]}]
|
||||||
|
except Exception as e:
|
||||||
|
findings = [{"severity": DRIFT_ERROR, "check": "probe", "detail": str(e)[:100]}]
|
||||||
|
for f in findings:
|
||||||
|
rows.append((name, f))
|
||||||
|
|
||||||
|
_print_canary_table(rows)
|
||||||
|
if any(f["severity"] == DRIFT_ERROR for _, f in rows):
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
def _print_canary_table(rows: list[tuple[str, dict]]) -> None:
|
||||||
|
table = Table(title="API Drift Canary", show_header=True)
|
||||||
|
table.add_column("Provider", style="bold")
|
||||||
|
table.add_column("Severity", justify="center")
|
||||||
|
table.add_column("Check")
|
||||||
|
table.add_column("Detail")
|
||||||
|
badge = {
|
||||||
|
DRIFT_OK: "[green]OK[/green]",
|
||||||
|
DRIFT_WARN: "[yellow]WARN[/yellow]",
|
||||||
|
DRIFT_ERROR: "[red]ERROR[/red]",
|
||||||
|
}
|
||||||
|
for name, f in rows:
|
||||||
|
table.add_row(
|
||||||
|
name,
|
||||||
|
badge.get(f["severity"], f["severity"]),
|
||||||
|
f.get("check", ""),
|
||||||
|
f.get("detail", ""),
|
||||||
|
)
|
||||||
|
console.print(table)
|
||||||
|
|
||||||
|
|
||||||
def _run_doctor_checks(cache: Cache | None = None) -> list[dict]:
|
def _run_doctor_checks(cache: Cache | None = None) -> list[dict]:
|
||||||
"""Run all doctor checks and return results.
|
"""Run all doctor checks and return results.
|
||||||
|
|
||||||
|
|||||||
@@ -50,6 +50,27 @@ VALID_HIDDEN_CONTENT_POLICIES = {
|
|||||||
HIDDEN_CONTENT_OMIT,
|
HIDDEN_CONTENT_OMIT,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# API-drift canary (FUTURE.md §10)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The export depends on undocumented internal web APIs that can change shape
|
||||||
|
# without notice. The canary fetches a small live sample and asserts only the
|
||||||
|
# normalizer's load-bearing fields (NOT full shape — provider settings objects
|
||||||
|
# churn constantly and would be pure noise). Findings carry a severity:
|
||||||
|
# ERROR — a load-bearing field is missing/mistyped; the parser will break or
|
||||||
|
# silently lose data. The backup is no longer trustworthy.
|
||||||
|
# WARN — something unfamiliar appeared (new content_type, new tool author,
|
||||||
|
# a dormant field went live). Investigate; not necessarily broken.
|
||||||
|
# OK — the assertion held.
|
||||||
|
DRIFT_ERROR = "error"
|
||||||
|
DRIFT_WARN = "warn"
|
||||||
|
DRIFT_OK = "ok"
|
||||||
|
|
||||||
|
|
||||||
|
def drift_finding(severity: str, check: str, detail: str = "") -> dict:
|
||||||
|
"""Build one canary finding."""
|
||||||
|
return {"severity": severity, "check": check, "detail": detail}
|
||||||
|
|
||||||
|
|
||||||
def resolve_hidden_content_policy() -> str:
|
def resolve_hidden_content_policy() -> str:
|
||||||
"""Read EXPORTER_HIDDEN_CONTENT from the environment, defaulting to placeholder."""
|
"""Read EXPORTER_HIDDEN_CONTENT from the environment, defaulting to placeholder."""
|
||||||
@@ -159,6 +180,15 @@ class BaseProvider(ABC):
|
|||||||
isolation, e.g. from tests, doesn't crash).
|
isolation, e.g. from tests, doesn't crash).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
def check_drift(self) -> list[dict]:
|
||||||
|
"""Probe the live API and assert the normalizer's load-bearing fields.
|
||||||
|
|
||||||
|
Returns a list of ``drift_finding`` dicts. The default is a no-op
|
||||||
|
(used by providers with no remote schema risk, e.g. local Claude Code
|
||||||
|
transcripts); web-API providers override this. See FUTURE.md §10.
|
||||||
|
"""
|
||||||
|
return []
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Concrete helpers
|
# Concrete helpers
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|||||||
@@ -45,12 +45,16 @@ from src.blocks import (
|
|||||||
from src.loss_report import LossReport
|
from src.loss_report import LossReport
|
||||||
from src.providers.base import (
|
from src.providers.base import (
|
||||||
BaseProvider,
|
BaseProvider,
|
||||||
|
DRIFT_ERROR,
|
||||||
|
DRIFT_OK,
|
||||||
|
DRIFT_WARN,
|
||||||
HIDDEN_CONTENT_FULL,
|
HIDDEN_CONTENT_FULL,
|
||||||
HIDDEN_CONTENT_OMIT,
|
HIDDEN_CONTENT_OMIT,
|
||||||
HIDDEN_CONTENT_PLACEHOLDER,
|
HIDDEN_CONTENT_PLACEHOLDER,
|
||||||
ProviderError,
|
ProviderError,
|
||||||
REQUEST_TIMEOUT,
|
REQUEST_TIMEOUT,
|
||||||
VALID_HIDDEN_CONTENT_POLICIES,
|
VALID_HIDDEN_CONTENT_POLICIES,
|
||||||
|
drift_finding,
|
||||||
resolve_hidden_content_policy,
|
resolve_hidden_content_policy,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -91,6 +95,20 @@ def parse_asset_file_id(ref: str) -> str | None:
|
|||||||
# myfiles_browser is the legacy name for the same retrieval tool.
|
# myfiles_browser is the legacy name for the same retrieval tool.
|
||||||
_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}
|
_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}
|
||||||
|
|
||||||
|
# ── API-drift canary vocabularies (FUTURE.md §10) ──────────────────────────
|
||||||
|
# Tool authors seen live 2026-06-28. The collapse keys off author.name, so a
|
||||||
|
# RENAMED retrieval tool would silently bypass the collapse and re-bloat the
|
||||||
|
# archive — the canary flags any unfamiliar (role="tool", author.name) pair.
|
||||||
|
_KNOWN_TOOL_AUTHORS = _COLLAPSE_TOOL_AUTHORS | {"web.run", "python"}
|
||||||
|
# content_types the normalizer dispatches on (keep in sync with
|
||||||
|
# _extract_blocks_for_content). A new value already degrades to an `unknown`
|
||||||
|
# block + LossReport at export time; the canary flags it proactively.
|
||||||
|
_HANDLED_CONTENT_TYPES = frozenset({
|
||||||
|
"text", "multimodal_text", "execution_output", "system_error",
|
||||||
|
"tether_browsing_display", "code", "thoughts", "reasoning_recap",
|
||||||
|
"user_editable_context", "model_editable_context", "image_asset_pointer",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
class ChatGPTProvider(BaseProvider):
|
class ChatGPTProvider(BaseProvider):
|
||||||
"""Provider for ChatGPT conversations via the internal web API.
|
"""Provider for ChatGPT conversations via the internal web API.
|
||||||
@@ -766,6 +784,84 @@ class ChatGPTProvider(BaseProvider):
|
|||||||
"messages": messages,
|
"messages": messages,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
def check_drift(self) -> list[dict]:
|
||||||
|
"""Probe one listing page + one conversation; assert load-bearing fields.
|
||||||
|
|
||||||
|
See FUTURE.md §10. Asserts only the fields the normalizer depends on —
|
||||||
|
not the full response shape (which churns harmlessly). The top silent
|
||||||
|
risk is a renamed retrieval tool author bypassing the collapse.
|
||||||
|
"""
|
||||||
|
findings: list[dict] = []
|
||||||
|
page = self.list_conversations(offset=0, limit=5)
|
||||||
|
if not page:
|
||||||
|
return [drift_finding(DRIFT_WARN, "listing",
|
||||||
|
"empty listing — cannot verify shape")]
|
||||||
|
|
||||||
|
item = page[0]
|
||||||
|
for f in ("id", "title"):
|
||||||
|
if f not in item:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "listing", f"summary item missing '{f}'"))
|
||||||
|
if not (item.get("update_time") or item.get("create_time")):
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "listing", "no update_time/create_time on summary"))
|
||||||
|
|
||||||
|
conv_id = item.get("id")
|
||||||
|
if not conv_id:
|
||||||
|
return findings or [drift_finding(DRIFT_ERROR, "listing", "no id to fetch")]
|
||||||
|
raw = self.get_conversation(conv_id)
|
||||||
|
if not (raw.get("conversation_id") or raw.get("id")):
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "detail", "no conversation_id/id on detail"))
|
||||||
|
mapping = raw.get("mapping")
|
||||||
|
if not isinstance(mapping, dict) or not mapping:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "detail", "mapping missing/empty — tree walk will yield 0 messages"))
|
||||||
|
return findings
|
||||||
|
|
||||||
|
saw_text_with_parts = False
|
||||||
|
seen_new_ct: set[str] = set()
|
||||||
|
seen_new_tool: set[str] = set()
|
||||||
|
for node in mapping.values():
|
||||||
|
if "children" not in node:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "mapping", "node missing 'children' — tree walk breaks"))
|
||||||
|
break
|
||||||
|
msg = node.get("message")
|
||||||
|
if not msg:
|
||||||
|
continue
|
||||||
|
author = msg.get("author") or {}
|
||||||
|
role, name = author.get("role"), author.get("name")
|
||||||
|
content = msg.get("content") or {}
|
||||||
|
ct = content.get("content_type")
|
||||||
|
if ct and ct not in _HANDLED_CONTENT_TYPES:
|
||||||
|
seen_new_ct.add(ct)
|
||||||
|
if role == "tool" and name and name not in _KNOWN_TOOL_AUTHORS:
|
||||||
|
seen_new_tool.add(name)
|
||||||
|
if ct == "text":
|
||||||
|
parts = content.get("parts") or []
|
||||||
|
if any(isinstance(p, str) and p.strip() for p in parts):
|
||||||
|
saw_text_with_parts = True
|
||||||
|
|
||||||
|
for name in sorted(seen_new_tool):
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "collapse",
|
||||||
|
f"unfamiliar tool author {name!r} — if it's a retrieval dump it "
|
||||||
|
"is NOT being collapsed (silent archive bloat); update "
|
||||||
|
"_COLLAPSE_TOOL_AUTHORS / _KNOWN_TOOL_AUTHORS"))
|
||||||
|
for ct in sorted(seen_new_ct):
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "content_type",
|
||||||
|
f"new content_type {ct!r} — exported as an unknown block; add a handler"))
|
||||||
|
if not saw_text_with_parts:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "parts",
|
||||||
|
"no text message yielded non-empty parts — content.parts may have drifted"))
|
||||||
|
|
||||||
|
if not findings:
|
||||||
|
findings.append(drift_finding(DRIFT_OK, "schema", "all load-bearing fields present"))
|
||||||
|
return findings
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Internal helpers
|
# Internal helpers
|
||||||
|
|||||||
+72
-1
@@ -16,7 +16,14 @@ from src.blocks import (
|
|||||||
make_unknown_block,
|
make_unknown_block,
|
||||||
)
|
)
|
||||||
from src.loss_report import LossReport
|
from src.loss_report import LossReport
|
||||||
from src.providers.base import BaseProvider, ProviderError
|
from src.providers.base import (
|
||||||
|
BaseProvider,
|
||||||
|
DRIFT_ERROR,
|
||||||
|
DRIFT_OK,
|
||||||
|
DRIFT_WARN,
|
||||||
|
ProviderError,
|
||||||
|
drift_finding,
|
||||||
|
)
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -232,6 +239,70 @@ class ClaudeProvider(BaseProvider):
|
|||||||
"messages": messages,
|
"messages": messages,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
def check_drift(self) -> list[dict]:
|
||||||
|
"""Probe one listing page + one conversation; assert load-bearing fields.
|
||||||
|
|
||||||
|
See FUTURE.md §10. Real Claude messages are flat ``text``/``sender``;
|
||||||
|
the rich ``content`` block path is dormant, so its activation (content
|
||||||
|
becoming a list) is itself a drift event worth flagging. ``attachments``
|
||||||
|
/ ``files`` are ignored by the normalizer — non-empty values are a
|
||||||
|
silent-loss risk.
|
||||||
|
"""
|
||||||
|
findings: list[dict] = []
|
||||||
|
page = self.list_conversations(offset=0, limit=5)
|
||||||
|
if not page:
|
||||||
|
return [drift_finding(DRIFT_WARN, "listing",
|
||||||
|
"empty listing — cannot verify shape")]
|
||||||
|
|
||||||
|
item = page[0]
|
||||||
|
if not (item.get("uuid") or item.get("id")):
|
||||||
|
findings.append(drift_finding(DRIFT_ERROR, "listing", "summary missing uuid/id"))
|
||||||
|
if not (item.get("name") or item.get("title")):
|
||||||
|
findings.append(drift_finding(DRIFT_WARN, "listing", "no name/title on summary"))
|
||||||
|
if not (item.get("updated_at") or item.get("update_time")):
|
||||||
|
findings.append(drift_finding(DRIFT_WARN, "listing", "no updated_at on summary"))
|
||||||
|
|
||||||
|
conv_id = item.get("uuid") or item.get("id")
|
||||||
|
if not conv_id:
|
||||||
|
return findings or [drift_finding(DRIFT_ERROR, "listing", "no id to fetch")]
|
||||||
|
raw = self.get_conversation(conv_id)
|
||||||
|
if not (raw.get("uuid") or raw.get("id")):
|
||||||
|
findings.append(drift_finding(DRIFT_ERROR, "detail", "no uuid/id on detail"))
|
||||||
|
msgs = raw.get("chat_messages") or raw.get("messages")
|
||||||
|
if not isinstance(msgs, list) or not msgs:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "detail", "chat_messages missing/empty — 0 messages"))
|
||||||
|
return findings
|
||||||
|
|
||||||
|
saw_sender = saw_text = False
|
||||||
|
flagged_list = flagged_attach = False
|
||||||
|
for m in msgs:
|
||||||
|
if m.get("sender") or m.get("role"):
|
||||||
|
saw_sender = True
|
||||||
|
content = m.get("content")
|
||||||
|
if isinstance(content, list) and not flagged_list:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "content",
|
||||||
|
"message.content is now a LIST — Claude switched to typed blocks; "
|
||||||
|
"the dormant _dispatch_claude_block path is now live, verify rendering"))
|
||||||
|
flagged_list = True
|
||||||
|
if m.get("text") or content:
|
||||||
|
saw_text = True
|
||||||
|
if (m.get("attachments") or m.get("files")) and not flagged_attach:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "attachments",
|
||||||
|
"message has non-empty attachments/files — the normalizer ignores "
|
||||||
|
"these (silent loss); add handling"))
|
||||||
|
flagged_attach = True
|
||||||
|
|
||||||
|
if not saw_sender:
|
||||||
|
findings.append(drift_finding(DRIFT_ERROR, "messages", "no sender/role on any message"))
|
||||||
|
if not saw_text:
|
||||||
|
findings.append(drift_finding(DRIFT_WARN, "messages", "no text/content found"))
|
||||||
|
if not findings:
|
||||||
|
findings.append(drift_finding(DRIFT_OK, "schema", "all load-bearing fields present"))
|
||||||
|
return findings
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Internal helpers
|
# Internal helpers
|
||||||
|
|||||||
@@ -275,3 +275,25 @@ class TestPrune:
|
|||||||
assert result.exit_code == 0
|
assert result.exit_code == 0
|
||||||
assert "Aborted" in result.output
|
assert "Aborted" in result.output
|
||||||
assert stale.exists()
|
assert stale.exists()
|
||||||
|
|
||||||
|
|
||||||
|
class TestCanaryCommand:
|
||||||
|
"""`canary` wiring — no live API calls (no tokens configured)."""
|
||||||
|
|
||||||
|
def test_no_tokens_exits_nonzero(self, tmp_path):
|
||||||
|
cache = Cache(tmp_path / "cache")
|
||||||
|
cache.acknowledge_tos()
|
||||||
|
runner = CliRunner(mix_stderr=True)
|
||||||
|
result = runner.invoke(
|
||||||
|
cli,
|
||||||
|
["--no-log-file", "canary"],
|
||||||
|
env={
|
||||||
|
"CACHE_DIR": str(tmp_path / "cache"),
|
||||||
|
"EXPORT_DIR": str(tmp_path / "exports"),
|
||||||
|
# Empty so .env (override=False) can't repopulate real tokens.
|
||||||
|
"CHATGPT_SESSION_TOKEN": "",
|
||||||
|
"CLAUDE_SESSION_KEY": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "No web-API provider tokens" in result.output
|
||||||
|
|||||||
@@ -973,3 +973,146 @@ class TestChatGPTSessionHealth:
|
|||||||
ok, detail = p.session_health()
|
ok, detail = p.session_health()
|
||||||
assert ok is False
|
assert ok is False
|
||||||
assert "unreachable" in detail.lower()
|
assert "unreachable" in detail.lower()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# API-drift canary (§10): assert load-bearing fields, flag silent risks.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _sev(findings):
|
||||||
|
return {f["severity"] for f in findings}
|
||||||
|
|
||||||
|
|
||||||
|
def _checks(findings, severity):
|
||||||
|
return {f["check"] for f in findings if f["severity"] == severity}
|
||||||
|
|
||||||
|
|
||||||
|
class TestChatGPTDriftCanary:
|
||||||
|
def _provider(self, page, detail):
|
||||||
|
from src.providers.chatgpt import ChatGPTProvider
|
||||||
|
|
||||||
|
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||||
|
p.list_conversations = lambda offset=0, limit=5: page
|
||||||
|
p.get_conversation = lambda cid: detail
|
||||||
|
return p
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _detail(mapping):
|
||||||
|
return {
|
||||||
|
"conversation_id": "c1", "title": "T",
|
||||||
|
"create_time": 1.0, "update_time": 2.0, "mapping": mapping,
|
||||||
|
}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _node(role="user", name=None, content_type="text", parts=("hello",)):
|
||||||
|
return {
|
||||||
|
"id": "n", "parent": None, "children": [],
|
||||||
|
"message": {
|
||||||
|
"author": {"role": role, "name": name},
|
||||||
|
"content": {"content_type": content_type, "parts": list(parts)},
|
||||||
|
"metadata": {},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_healthy_shape_is_ok_only(self):
|
||||||
|
from src.providers.base import DRIFT_OK
|
||||||
|
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail({"n1": self._node()}))
|
||||||
|
assert _sev(p.check_drift()) == {DRIFT_OK}
|
||||||
|
|
||||||
|
def test_missing_listing_id_is_error(self):
|
||||||
|
from src.providers.base import DRIFT_ERROR
|
||||||
|
|
||||||
|
p = self._provider([{"title": "T"}], self._detail({"n1": self._node()}))
|
||||||
|
assert DRIFT_ERROR in _sev(p.check_drift())
|
||||||
|
|
||||||
|
def test_empty_mapping_is_error(self):
|
||||||
|
from src.providers.base import DRIFT_ERROR
|
||||||
|
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail({}))
|
||||||
|
findings = p.check_drift()
|
||||||
|
assert "detail" in _checks(findings, DRIFT_ERROR)
|
||||||
|
|
||||||
|
def test_unfamiliar_tool_author_warns_collapse(self):
|
||||||
|
from src.providers.base import DRIFT_WARN
|
||||||
|
|
||||||
|
mapping = {
|
||||||
|
"n1": self._node(), # a healthy text msg so parts check passes
|
||||||
|
"n2": self._node(role="tool", name="brand_new_retriever"),
|
||||||
|
}
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail(mapping))
|
||||||
|
assert "collapse" in _checks(p.check_drift(), DRIFT_WARN)
|
||||||
|
|
||||||
|
def test_new_content_type_warns(self):
|
||||||
|
from src.providers.base import DRIFT_WARN
|
||||||
|
|
||||||
|
mapping = {
|
||||||
|
"n1": self._node(),
|
||||||
|
"n2": self._node(content_type="hologram_v2", parts=()),
|
||||||
|
}
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail(mapping))
|
||||||
|
assert "content_type" in _checks(p.check_drift(), DRIFT_WARN)
|
||||||
|
|
||||||
|
def test_known_tool_author_not_flagged(self):
|
||||||
|
# file_search is a known collapse author — must NOT warn.
|
||||||
|
mapping = {
|
||||||
|
"n1": self._node(),
|
||||||
|
"n2": self._node(role="tool", name="file_search"),
|
||||||
|
}
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail(mapping))
|
||||||
|
assert "collapse" not in _checks(p.check_drift(), "warn")
|
||||||
|
|
||||||
|
|
||||||
|
class TestClaudeDriftCanary:
|
||||||
|
def _provider(self, page, detail):
|
||||||
|
from src.providers.claude import ClaudeProvider
|
||||||
|
|
||||||
|
p = ClaudeProvider.__new__(ClaudeProvider)
|
||||||
|
p.list_conversations = lambda offset=0, limit=5: page
|
||||||
|
p.get_conversation = lambda cid: detail
|
||||||
|
return p
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _detail(msgs):
|
||||||
|
return {"uuid": "u1", "name": "N", "created_at": "x", "updated_at": "y",
|
||||||
|
"chat_messages": msgs}
|
||||||
|
|
||||||
|
def test_healthy_flat_shape_is_ok(self):
|
||||||
|
from src.providers.base import DRIFT_OK
|
||||||
|
|
||||||
|
msgs = [{"uuid": "m1", "sender": "human", "text": "hi",
|
||||||
|
"created_at": "x", "attachments": [], "files": []}]
|
||||||
|
p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
|
||||||
|
self._detail(msgs))
|
||||||
|
assert _sev(p.check_drift()) == {DRIFT_OK}
|
||||||
|
|
||||||
|
def test_content_as_list_warns(self):
|
||||||
|
from src.providers.base import DRIFT_WARN
|
||||||
|
|
||||||
|
msgs = [{"uuid": "m1", "sender": "assistant",
|
||||||
|
"content": [{"type": "text", "text": "hi"}], "created_at": "x"}]
|
||||||
|
p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
|
||||||
|
self._detail(msgs))
|
||||||
|
assert "content" in _checks(p.check_drift(), DRIFT_WARN)
|
||||||
|
|
||||||
|
def test_nonempty_attachments_warn(self):
|
||||||
|
from src.providers.base import DRIFT_WARN
|
||||||
|
|
||||||
|
msgs = [{"uuid": "m1", "sender": "human", "text": "hi",
|
||||||
|
"created_at": "x", "attachments": [{"file_name": "a.pdf"}]}]
|
||||||
|
p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
|
||||||
|
self._detail(msgs))
|
||||||
|
assert "attachments" in _checks(p.check_drift(), DRIFT_WARN)
|
||||||
|
|
||||||
|
def test_empty_messages_is_error(self):
|
||||||
|
from src.providers.base import DRIFT_ERROR
|
||||||
|
|
||||||
|
p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
|
||||||
|
self._detail([]))
|
||||||
|
assert DRIFT_ERROR in _sev(p.check_drift())
|
||||||
|
|||||||
Reference in New Issue
Block a user