Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1f5a445ada | ||
|
|
bbcb29c856 | ||
|
|
1e016ea652 | ||
|
|
ef603cf659 | ||
|
|
4a7ee5773f | ||
|
|
3cf0b1eaa8 | ||
|
|
456975ad50 |
@@ -26,6 +26,18 @@ CHATGPT_PROJECT_IDS=
|
|||||||
# Token type: opaque string. Typically valid for ~30 days.
|
# Token type: opaque string. Typically valid for ~30 days.
|
||||||
CLAUDE_SESSION_KEY=
|
CLAUDE_SESSION_KEY=
|
||||||
|
|
||||||
|
# --- Claude Code (local agent sessions) ---
|
||||||
|
# The claude-code provider reads local Claude Code transcripts. By default it
|
||||||
|
# scans ~/.claude/projects/ (plus $CLAUDE_CONFIG_DIR/projects when that is set).
|
||||||
|
# To scan additional roots — e.g. other machines' sessions copied onto this box —
|
||||||
|
# set a ':'-separated list of projects roots. Sessions are merged by folder.
|
||||||
|
#CLAUDE_CODE_DIR=~/.claude/projects:/mnt/backup/laptop/.claude/projects
|
||||||
|
#
|
||||||
|
# Session titles are tagged with the git repos they touched, e.g.
|
||||||
|
# "Resume StartWRT work [start-technologies]". To never tag specific repos,
|
||||||
|
# list their names here (comma-separated).
|
||||||
|
#CLAUDE_CODE_REPO_TAG_IGNORE=some-repo,another-repo
|
||||||
|
|
||||||
# --- Output ---
|
# --- Output ---
|
||||||
# Where exported Markdown files are written (default: ./exports)
|
# Where exported Markdown files are written (default: ./exports)
|
||||||
EXPORT_DIR=./exports
|
EXPORT_DIR=./exports
|
||||||
|
|||||||
@@ -3,6 +3,34 @@
|
|||||||
All notable changes to this project will be documented here.
|
All notable changes to this project will be documented here.
|
||||||
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||||
|
|
||||||
|
## [0.8.0] - 2026-07-06
|
||||||
|
|
||||||
|
Focused on the Claude Code provider after the tool's on-disk layout changed and
|
||||||
|
Claude Code sessions proved hard to find in Joplin.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- **Subagent capture.** Claude Code now stores subagent (Task-tool) transcripts as separate `<session>/subagents/agent-*.jsonl` files (with an `agent-*.meta.json` sidecar). These were previously invisible to the exporter — all delegated work (reviews, research, plans) was lost. Each subagent is now folded into its parent session inline at the `Task`/`Agent` call that spawned it, rendered as a collapsible `<details>` block labeled with its `agentType`/`description`. The subagent's own tool traffic is collapsed under the same `EXPORTER_HIDDEN_CONTENT` policy as the main dialogue; nesting is handled at any depth via `toolUseId` matching.
|
||||||
|
- **Repo tags in Claude Code titles.** Sessions launched from a workspace root all land in one folder-named notebook, so titles now carry the repos each session touched, e.g. `Resume StartWRT project work [start-technologies]` — visible in the note list and matched by Joplin search. A file's repo is the git repository it lives in (nearest ancestor with a `.git`), resolved from tool_use paths anywhere in the filesystem — so cross-workspace work is captured and non-repo noise (config dirs, one-off files, reference dirs) is excluded because it isn't a git repo. Frequency-ordered, capped at 3. Escape hatch: `CLAUDE_CODE_REPO_TAG_IGNORE` (comma-separated repo names). Tags reflect the current git layout, so a since-deleted/moved repo drops from the tag on re-export.
|
||||||
|
- **Multiple projects roots.** `CLAUDE_CODE_DIR` now accepts an `os.pathsep`-separated list of roots (a single path stays backward compatible), and `CLAUDE_CONFIG_DIR`'s `projects/` tree is scanned automatically when set. Sessions from all roots are merged by launch-folder; a session UUID present in two roots keeps the newer-mtime copy.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- **Claude Code gets its own top-level Joplin notebook, `AI-ClaudeCode`** (was nested under `AI-Claude` alongside Claude web chats, which made dev sessions hard to find). Existing Claude Code notes self-heal into the new notebook on the next `joplin` run — `update_note` now sets `parent_id`, so a changed provider→notebook mapping relocates notes in place instead of duplicating them. After migrating, the emptied `AI-Claude/{Myworkspace,Services,…}` notebooks can be deleted by hand (Joplin does not auto-remove empty folders).
|
||||||
|
|
||||||
|
### Notes
|
||||||
|
- The sibling `<session>/tool-results/*.txt` sidecars (externalized large tool outputs) are intentionally not captured — tool_result content is collapsed under the default policy anyway.
|
||||||
|
|
||||||
|
## [0.7.0] - 2026-06-28
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- `canary` command — probes the ChatGPT/Claude web APIs for schema drift against the fields the parser actually depends on (one listing page + one conversation per provider), not the full response shape. Flags the silent-failure risks for a backup tool: a renamed retrieval-tool author bypassing the hidden-content collapse, a new `content_type`, drifted message fields, or non-empty Claude `attachments`/`files` the normalizer ignores. ERROR findings exit non-zero; WARN findings are surfaced but non-fatal. See `FUTURE.md` §10.
|
||||||
|
- `doctor` now reports a real ChatGPT token-health check ("ChatGPT token active") based on the `/api/auth/session` `error` field. ChatGPT session tokens are JWEs whose `exp` is encrypted and unreadable client-side; the previous decode path could never yield an expiry. The honest signal is `error == "RefreshAccessTokenError"` (verified live), which means the session token is dead even though `expires`/`accessToken` still echo stale values. See `FUTURE.md` §9.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- `ChatGPTProvider._fetch_access_token` now checks the `/api/auth/session` `error` field and fails fast with a clear "refresh your token" message. Previously it returned the stale `accessToken` present on a dead session, producing a confusing downstream 401 instead of an actionable auth error.
|
||||||
|
|
||||||
|
### Removed
|
||||||
|
- `auth --from-browser` and the `browser-cookie3` dependency (`src/browser_tokens.py`). Browser cookie auto-extraction is not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave) keys cookies off a SYSTEM-level layer that can't be decrypted off disk without Administrator rights and AV-flagged SYSTEM impersonation, and fails on Brave specifically (symptom: `Unable to get key for cookie decryption`). Auth is manual (DevTools wizard) only. See `FUTURE.md` §2 for the full rationale.
|
||||||
|
|
||||||
## [0.6.0] - 2026-06-12
|
## [0.6.0] - 2026-06-12
|
||||||
|
|
||||||
First tagged release. The project was developed through several internal
|
First tagged release. The project was developed through several internal
|
||||||
@@ -46,6 +74,11 @@ consolidated here.
|
|||||||
- Session limiter: `--max-conversations N` / `MAX_CONVERSATIONS_PER_RUN` cap downloads per run (per provider); capped-out conversations are reported as "deferred" and resumed next run.
|
- Session limiter: `--max-conversations N` / `MAX_CONVERSATIONS_PER_RUN` cap downloads per run (per provider); capped-out conversations are reported as "deferred" and resumed next run.
|
||||||
- Polite pacing: `REQUEST_DELAY` (default 1.0s, ±25% jitter, `0` disables) between consecutive API requests, with the existing 429 backoff as the reactive net.
|
- Polite pacing: `REQUEST_DELAY` (default 1.0s, ±25% jitter, `0` disables) between consecutive API requests, with the existing 429 backoff as the reactive net.
|
||||||
|
|
||||||
|
**Re-rendering**
|
||||||
|
- `export --force` re-exports every conversation even if cached and unchanged, so the whole archive can be re-rendered after a formatting/feature change without `cache --clear`. Runs as a tracked campaign (a stamped start time in the manifest): each run re-renders the least-recently-exported conversations, the "still to go" count shrinks toward zero, finished providers do no further work, and the campaign auto-completes (reporting "Force re-render complete"). Combines with `--max-conversations` to spread the work across runs.
|
||||||
|
- `mark_exported` now preserves `joplin_note_id` / `joplin_synced_at` / `joplin_resources` across re-exports (refreshing `exported_at`), so a re-export followed by `joplin` updates the existing notes instead of creating duplicates.
|
||||||
|
- Fix: `.env` is now loaded at the very start of every command, so `CACHE_DIR` and `LOG_FILE` are honored (previously the cache silently used `./cache` regardless of `CACHE_DIR`).
|
||||||
|
|
||||||
**Archive hygiene**
|
**Archive hygiene**
|
||||||
- `prune` deletes export files not referenced by the manifest (old-layout trees, no-ID orphans), with listing, confirmation, `--dry-run`, `-y`, and empty-directory sweep. Refuses to run when the manifest references no files (so `cache --clear` + `prune` can't wipe the archive). First live run removed 420 stale files (9.4 MB).
|
- `prune` deletes export files not referenced by the manifest (old-layout trees, no-ID orphans), with listing, confirmation, `--dry-run`, `-y`, and empty-directory sweep. Refuses to run when the manifest references no files (so `cache --clear` + `prune` can't wipe the archive). First live run removed 420 stale files (9.4 MB).
|
||||||
- `doctor` verifies manifest ↔ disk integrity: every recorded `file_path` must exist on disk.
|
- `doctor` verifies manifest ↔ disk integrity: every recorded `file_path` must exist on disk.
|
||||||
|
|||||||
@@ -1,8 +1,21 @@
|
|||||||
# Planned Future Work
|
# Planned Future Work
|
||||||
|
|
||||||
Items completed in each release are moved to the changelog. Items here are
|
> **Status 2026-07-06 (v0.8.0): Claude Code coverage reopened and shipped.**
|
||||||
designed for but not yet implemented. The codebase is structured to make each
|
> Claude Code changed its on-disk layout (subagent transcripts moved to separate
|
||||||
of these additions straightforward.
|
> `subagents/*.jsonl` files) and its sessions were hard to find in Joplin. v0.8.0
|
||||||
|
> addressed this: subagent capture (folded `<details>`), repo `[tags]` in titles,
|
||||||
|
> an own `AI-ClaudeCode` notebook with self-healing note moves, and multi-root
|
||||||
|
> scanning (`CLAUDE_CODE_DIR` list + `CLAUDE_CONFIG_DIR`). See the changelog.
|
||||||
|
>
|
||||||
|
> **Status 2026-06-28: feature-complete / done for now.** As of v0.7.0 the
|
||||||
|
> active roadmap is empty and the remaining backlog below has been **closed as
|
||||||
|
> not needed** — the tool does what it's needed to do as a local, manually-run
|
||||||
|
> backup CLI. Items are kept for reference only; revisit on demand if a real
|
||||||
|
> need shows up. Nothing here is planned work.
|
||||||
|
|
||||||
|
Items completed in each release are moved to the changelog. Items below the
|
||||||
|
roadmap were designed for but intentionally not implemented. The codebase is
|
||||||
|
structured to make each of these additions straightforward if ever revived.
|
||||||
|
|
||||||
**Completed:**
|
**Completed:**
|
||||||
- v0.1.0 — Core export: ChatGPT + Claude, incremental sync, Markdown + JSON output
|
- v0.1.0 — Core export: ChatGPT + Claude, incremental sync, Markdown + JSON output
|
||||||
@@ -17,14 +30,18 @@ of these additions straightforward.
|
|||||||
|
|
||||||
Priorities reflect the tool's primary purpose — a trustworthy backup so that
|
Priorities reflect the tool's primary purpose — a trustworthy backup so that
|
||||||
conversation data is not lost if a provider account is ever closed — plus the
|
conversation data is not lost if a provider account is ever closed — plus the
|
||||||
day-to-day friction of weekly ChatGPT token refresh, and the long-term goal
|
day-to-day friction of the weekly ChatGPT token refresh. The tool stays a
|
||||||
of running headless as a StartOS service.
|
local, manually-run CLI; the headless/StartOS direction was dropped
|
||||||
|
2026-06-28 (see #7 and #8), which also retires the token-freshness problem
|
||||||
|
(manual refresh is sufficient).
|
||||||
|
|
||||||
**Now (in order):**
|
**Now (in order):**
|
||||||
1. ~~Collapse tool retrieval dumps & hidden context~~ — **shipped in v0.6.0**
|
1. ~~Collapse tool retrieval dumps & hidden context~~ — **shipped in v0.6.0**
|
||||||
(full-archive re-export still pending; see entry below)
|
(full-archive `export --force` re-export completed 2026-06-13)
|
||||||
2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0** (note: runs on
|
2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0, removed afterward.**
|
||||||
whichever machine hosts the browser — Brave is not on this Linux box)
|
Not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave)
|
||||||
|
needs admin + SYSTEM impersonation that AV flags as credential theft, and
|
||||||
|
fails on Brave specifically. Auth is manual (DevTools) — see entry below.
|
||||||
3. ~~Claude Code session provider~~ — **shipped in v0.6.0**
|
3. ~~Claude Code session provider~~ — **shipped in v0.6.0**
|
||||||
4. ~~Archive hygiene — `prune` + `doctor` integrity check~~ — **shipped in v0.6.0**
|
4. ~~Archive hygiene — `prune` + `doctor` integrity check~~ — **shipped in v0.6.0**
|
||||||
|
|
||||||
@@ -32,12 +49,23 @@ of running headless as a StartOS service.
|
|||||||
|
|
||||||
5. ~~Binary content downloads~~ — **shipped in v0.6.0**
|
5. ~~Binary content downloads~~ — **shipped in v0.6.0**
|
||||||
6. ~~Per-session download limiter + polite pacing~~ — **shipped in v0.6.0**
|
6. ~~Per-session download limiter + polite pacing~~ — **shipped in v0.6.0**
|
||||||
7. Scheduled / watch mode → StartOS service packaging (long-term destination)
|
7. ~~Scheduled / watch mode~~ — **dropped 2026-06-28**; the tool stays a
|
||||||
|
manually-run CLI, so no in-app polling loop is needed
|
||||||
|
8. ~~StartOS service packaging~~ — **dropped 2026-06-28**; the local CLI is
|
||||||
|
sufficient (source convos live in the cloud and can be re-downloaded;
|
||||||
|
Joplin already syncs encrypted to an offsite S3 provider). Dropping this
|
||||||
|
also retires the headless token-freshness problem — manual weekly refresh
|
||||||
|
is fine.
|
||||||
|
|
||||||
|
**Active:**
|
||||||
|
9. ~~Surface remaining token validity on `doctor`~~ — **IMPLEMENTED 2026-06-28**
|
||||||
|
via the `/api/auth/session` `error` field (not `expires`). See §9.
|
||||||
|
10. ~~Provider API-drift detection~~ — **IMPLEMENTED 2026-06-28** as the
|
||||||
|
`canary` command. See §10.
|
||||||
|
|
||||||
**Deprioritized** (entries kept at the bottom of this file; revisit on
|
**Deprioritized** (entries kept at the bottom of this file; revisit on
|
||||||
demand): `--force` flags, per-conversation cache reset, official export-ZIP
|
demand): `--force` flags, per-conversation cache reset, official export-ZIP
|
||||||
fallback, o1/o3 reasoning reclassification, Obsidian output, token expiry
|
fallback, o1/o3 reasoning reclassification, Obsidian output, search command.
|
||||||
notifications (largely superseded by cookie auto-extraction), search command.
|
|
||||||
Additional web providers (Gemini/Grok/Perplexity) are explicitly out of
|
Additional web providers (Gemini/Grok/Perplexity) are explicitly out of
|
||||||
scope — no significant usage to archive.
|
scope — no significant usage to archive.
|
||||||
|
|
||||||
@@ -50,8 +78,8 @@ from live recon: the retrieval dumps are NOT flagged
|
|||||||
`is_visually_hidden_from_conversation` — `author.name == "file_search"` is
|
`is_visually_hidden_from_conversation` — `author.name == "file_search"` is
|
||||||
the discriminator (the hidden flag only marks Custom Instructions and small
|
the discriminator (the hidden flag only marks Custom Instructions and small
|
||||||
system stubs). Verified live: worst files shrink 93% (524KB → 36KB).
|
system stubs). Verified live: worst files shrink 93% (524KB → 36KB).
|
||||||
Remaining step: full-archive re-export (`cache --clear` + `export`) +
|
Full-archive re-export (the `export --force` campaign) + Joplin re-sync
|
||||||
Joplin re-sync, at the user's chosen time.
|
completed 2026-06-13.
|
||||||
|
|
||||||
**Problem (measured 2026-06-12 against a full fresh export):** 45% of the
|
**Problem (measured 2026-06-12 against a full fresh export):** 45% of the
|
||||||
entire 11.2MB archive (260 files) is tool-role messages; 29 files are
|
entire 11.2MB archive (260 files) is tool-role messages; 29 files are
|
||||||
@@ -89,37 +117,38 @@ file contains zero hidden-context blocks.
|
|||||||
Re-export workflow after shipping: `cache --clear` + `export` (same as
|
Re-export workflow after shipping: `cache --clear` + `export` (same as
|
||||||
the v0.4.0 migration).
|
the v0.4.0 migration).
|
||||||
|
|
||||||
## 2. Brave Cookie Auto-Extraction — SHIPPED v0.6.0
|
## 2. Brave Cookie Auto-Extraction — REMOVED (not viable)
|
||||||
|
|
||||||
**Implemented 2026-06-12**: `auth --from-browser [browser]`
|
**Shipped v0.6.0 (2026-06-12), removed 2026-06-27.** `auth --from-browser`
|
||||||
(`src/browser_tokens.py`, browser-cookie3 dependency, defaults to brave;
|
plus `src/browser_tokens.py` and the `browser-cookie3` dependency are gone.
|
||||||
chrome/chromium/edge/firefox also supported). Extracted tokens are validated
|
Auth is manual (DevTools) only. Do not re-attempt without a fundamentally
|
||||||
against the live API before `.env` is touched. **Discovered during
|
different mechanism (see below).
|
||||||
implementation: Brave is not installed on this machine** — the logged-in
|
|
||||||
browser lives elsewhere, so the flag only helps when the exporter runs on
|
|
||||||
that machine (verified the graceful-failure path here; the happy path is
|
|
||||||
covered by mocked tests). This strengthens the case for the StartOS
|
|
||||||
token-push mechanism (entry 8).
|
|
||||||
|
|
||||||
Pull session tokens directly from the Brave browser profile instead of the
|
**Why it doesn't work.** Modern Chromium browsers encrypt cookies on Windows
|
||||||
weekly manual DevTools dance. `auth` gains an "extract from browser" path
|
with **App-Bound Encryption** (Chrome 127+, July 2024; current Brave).
|
||||||
(with the manual flow kept as fallback); a later iteration could let
|
Cookies are written with a `v20` prefix and keyed off a secret wrapped in a
|
||||||
`doctor` or a 401 handler suggest/perform a re-extract automatically.
|
**SYSTEM-level** DPAPI layer plus app validation. `browser-cookie3` only
|
||||||
|
knows the legacy `v10`/DPAPI key, so its AES-GCM MAC check fails — the exact
|
||||||
|
symptom hit in the field:
|
||||||
|
|
||||||
- Brave on Linux follows the Chromium pattern: cookies SQLite at
|
```
|
||||||
`~/.config/BraveSoftware/Brave-Browser/Default/Cookies`, values
|
ChatGPT: Could not read brave cookies for chatgpt.com: Unable to get key for cookie decryption.
|
||||||
AES-128-CBC encrypted with a key derived from the OS keyring secret
|
```
|
||||||
("Brave Safe Storage" via SecretService/kwallet, or the `peanuts` v10
|
|
||||||
fallback). Well-trodden territory — evaluate a small dependency
|
Decrypting `v20` at all requires unwrapping the SYSTEM layer, which means
|
||||||
(`browser_cookie3`-style) vs. a focused in-repo implementation.
|
running as SYSTEM (e.g. a PsExec-style service) — i.e. **Administrator
|
||||||
- Extract `__Secure-next-auth.session-token.0` / `.1` from `chatgpt.com`
|
rights** and behavior that AV/EDR flags as infostealer activity. The one
|
||||||
and `sessionKey` from `claude.ai`; write to `.env` through the existing
|
maintained Python option (`rookiepy`) needs admin from Chrome v130+, was
|
||||||
auth-wizard write path (tokens never echoed).
|
**archived 2026-06-07**, and has an unresolved bug where **Brave returns 0
|
||||||
- Brave must be the user's logged-in browser (it is); detect a locked /
|
cookies** even after the key is retrieved. ABE is *designed* to stop exactly
|
||||||
unreadable cookie DB (Brave running with the DB exclusively locked) and
|
this, so no off-disk reader is a reliable, non-invasive fit.
|
||||||
fall back to the manual flow with a clear message.
|
|
||||||
- Does **not** solve headless/StartOS — no browser on the server. That
|
**If ever revisited:** the only non-admin path is Chrome Remote Debugging
|
||||||
needs a token-push mechanism, tracked under the StartOS entry.
|
(launch the browser with `--remote-debugging-port`, read cookies via
|
||||||
|
`Network.getAllCookies` — the running browser decrypts for you). Heavier and
|
||||||
|
intrusive; not worth it for a weekly token refresh that takes 30 seconds by
|
||||||
|
hand. With the headless/StartOS direction dropped (#8), manual DevTools
|
||||||
|
refresh is the accepted approach — no automated extraction is needed.
|
||||||
|
|
||||||
## 3. Claude Code Session Provider — SHIPPED v0.6.0
|
## 3. Claude Code Session Provider — SHIPPED v0.6.0
|
||||||
|
|
||||||
@@ -257,60 +286,193 @@ Note: the conversation *listing* (paginated, 100/page) still runs in full
|
|||||||
each time so the cache comparison works — the cap applies to the heavy
|
each time so the cache comparison works — the cap applies to the heavy
|
||||||
per-conversation detail fetches, which dominate request volume.
|
per-conversation detail fetches, which dominate request volume.
|
||||||
|
|
||||||
Together with watch mode, this is a stepping stone to the StartOS service:
|
This is a stepping stone to the StartOS service: a capped, politely-paced
|
||||||
a scheduled, capped, politely-paced export is the traffic profile a
|
export — scheduled by the host (cron/StartOS), not an in-app loop — is the
|
||||||
headless deployment needs.
|
traffic profile a headless deployment needs.
|
||||||
|
|
||||||
## 7. Scheduled / Watch Mode
|
## 7. Scheduled / Watch Mode — DROPPED (2026-06-28)
|
||||||
|
|
||||||
Add a `watch` command (or cron integration helper) to run exports automatically
|
An in-app `watch`/scheduler loop is not worth building. Scheduling belongs to
|
||||||
on a schedule:
|
whatever hosts the tool: a user cron line locally, and on the long-term
|
||||||
|
StartOS target the platform's own scheduling. Either way the tool only needs
|
||||||
|
to do one capped, politely-paced `export` + `joplin` run and exit — which it
|
||||||
|
already does. If cron ergonomics ever feel clunky, a thin `sync` subcommand
|
||||||
|
that chains `export` then `joplin` for a single cron line is a trivial
|
||||||
|
add-on, but the polling loop itself is off the roadmap.
|
||||||
|
|
||||||
```bash
|
## 8. StartOS Service Packaging — DROPPED (2026-06-28)
|
||||||
python -m src.main watch --interval 6h # poll every 6 hours
|
|
||||||
```
|
|
||||||
|
|
||||||
This would run `export` + `joplin` in sequence, then sleep. Alternatively,
|
Not pursuing a headless StartOS service. The local, manually-run CLI is
|
||||||
provide a `cron` command that prints the correct crontab line for the user's
|
sufficient: the source conversations live in the providers' clouds and can be
|
||||||
setup.
|
re-downloaded, and Joplin already syncs (encrypted) to an offsite S3 provider,
|
||||||
|
so durability is covered without a server in the loop.
|
||||||
|
|
||||||
Implementation: simple loop with `time.sleep()`, or emit a crontab entry
|
Dropping this also retires the one genuinely hard sub-problem it carried —
|
||||||
string that calls the export and joplin commands in sequence. A `--once`
|
session-token freshness without a browser. There is no headless context to
|
||||||
flag would do a single run then exit (useful for cron itself).
|
keep fresh; the weekly manual DevTools refresh is acceptable. (Local cookie
|
||||||
|
extraction remains a dead end regardless — see #2.)
|
||||||
|
|
||||||
Stepping stone to the StartOS service (below).
|
## 9. Token Validity on `doctor` — IMPLEMENTED (2026-06-28)
|
||||||
|
|
||||||
## 8. StartOS Service Packaging (long-term destination)
|
Shipped: `doctor` now adds a "ChatGPT token active" check via
|
||||||
|
`ChatGPTProvider.session_health()` (reads `/api/auth/session`, passes iff
|
||||||
|
`error` is falsy and `accessToken` is present), the never-working JWE/`exp`
|
||||||
|
decode path was removed, and `_fetch_access_token` now fails fast on a set
|
||||||
|
`error` instead of returning a stale token. Tests in
|
||||||
|
`tests/test_providers.py::TestChatGPTSessionHealth`. Investigation trail
|
||||||
|
below for the record.
|
||||||
|
|
||||||
Run the exporter headless as a StartOS service: scheduled export + sync to
|
Goal: if it's cheap to tell how much longer a token will work, show it on
|
||||||
a Joplin Server instance, so the backup happens without a desktop in the
|
`doctor`. Findings from live recon:
|
||||||
loop.
|
|
||||||
|
|
||||||
- Building blocks needed first: watch mode (#7), session limiter (#6),
|
- **Not readable from the token itself.** ChatGPT's `CHATGPT_SESSION_TOKEN`
|
||||||
and a Joplin Server (vs. desktop Web Clipper) sync target.
|
is a **JWE** (header `{"alg":"dir","enc":"A256GCM"}`, `eyJ…` prefix is just
|
||||||
- **Hard problem, flagged early:** session-token freshness without a
|
the encrypted protected header) — the `exp` claim is AES-256-GCM encrypted
|
||||||
browser. ChatGPT tokens last ~7 days; there is no browser on the server
|
with an OpenAI-only key, so it cannot be decoded client-side. Claude's
|
||||||
to re-extract from. Candidate mechanisms: a service config action the
|
`sk-…` key is fully opaque. The existing `doctor` JWT-decode path therefore
|
||||||
user pastes fresh tokens into periodically, or a future browser
|
never yields an expiry for the real tokens (falls to the "not decodable"
|
||||||
extension that pushes tokens to the server. Cookie auto-extraction (#2)
|
branch).
|
||||||
only solves the desktop case.
|
- **`/api/auth/session` exposes an `expires`** (the provider already calls
|
||||||
|
this endpoint in `_fetch_access_token`; the response includes `expires`
|
||||||
|
alongside `accessToken`). Live value observed 2026-06-28:
|
||||||
|
`2026-09-26` — **~90 days out**. This contradicts both the code's ~7-day
|
||||||
|
assumption and the lived weekly-refresh cadence, so it is almost certainly
|
||||||
|
the NextAuth **rolling session window** (re-extended on every call), not
|
||||||
|
the point at which the pasted token actually 401s. Displaying it verbatim
|
||||||
|
would give false confidence.
|
||||||
|
- **RESOLVED 2026-06-28 by a live 401 data point.** When the ChatGPT token
|
||||||
|
was actually dead (conversations API → 401), `/api/auth/session` still
|
||||||
|
returned **HTTP 200** with `expires: 2026-09-26` (~90 days out) — and that
|
||||||
|
`expires` *advanced* between two calls seconds apart (`04:20:08` → `04:30:37`).
|
||||||
|
So `expires` is a **rolling session window that rolls forward on every call
|
||||||
|
even for a dead token**; displaying it would actively lie. The same response
|
||||||
|
carried `error: "RefreshAccessTokenError"` and a stale `accessToken`.
|
||||||
|
- **The real signal is `error`, not `expires` or `accessToken`.** On a healthy
|
||||||
|
token `error` is absent/null; when the session token is dead NextAuth can't
|
||||||
|
refresh and sets `error: "RefreshAccessTokenError"` while still echoing a
|
||||||
|
rolling `expires` and a stale `accessToken`. This is an exact, free, binary
|
||||||
|
health check.
|
||||||
|
- **Design (ready to build):**
|
||||||
|
1. `doctor` ChatGPT check → read `/api/auth/session`; pass iff `error` is
|
||||||
|
falsy and `accessToken` present; on `RefreshAccessTokenError` report
|
||||||
|
"token expired — refresh". Drop the JWE/`exp` decode path (it can never
|
||||||
|
work) and do NOT surface `expires`.
|
||||||
|
2. Latent bug to fix alongside: `_fetch_access_token` reads `accessToken`
|
||||||
|
without checking `error`, so it proceeds with a stale token and yields a
|
||||||
|
confusing downstream 401 instead of a clear "refresh your token" message.
|
||||||
|
Check `error` there and fail fast.
|
||||||
|
3. Claude stays a 401-only signal (opaque `sk-`, no equivalent endpoint).
|
||||||
|
|
||||||
|
## 10. Provider API-Drift Detection — IMPLEMENTED (2026-06-28)
|
||||||
|
|
||||||
|
Shipped: the `canary` command + `BaseProvider.check_drift()` (overridden by
|
||||||
|
ChatGPT and Claude). It fetches one listing page + one conversation per
|
||||||
|
provider and asserts only the normalizer's load-bearing fields, emitting
|
||||||
|
`DRIFT_OK/WARN/ERROR` findings (`src/providers/base.py`). Severity badges
|
||||||
|
print as a Rich table; ERROR exits non-zero, WARN is non-fatal so a backup
|
||||||
|
run is never blocked. Drift vocabularies (`_KNOWN_TOOL_AUTHORS`,
|
||||||
|
`_HANDLED_CONTENT_TYPES`) live in `chatgpt.py` next to the collapse set they
|
||||||
|
guard. Tests: `TestChatGPTDriftCanary`, `TestClaudeDriftCanary`,
|
||||||
|
`TestCanaryCommand`. Verified live 2026-06-28 — both providers OK. Recon
|
||||||
|
trail below for the record.
|
||||||
|
|
||||||
|
## 10b. Provider API-Drift Detection — investigation (2026-06-28)
|
||||||
|
|
||||||
|
The export depends on undocumented internal web APIs (ChatGPT/Claude) that can
|
||||||
|
change shape without notice. The worst failure for a backup tool is *silent*:
|
||||||
|
a response-schema change that makes the exporter skip or mis-parse content
|
||||||
|
without erroring. `doctor` currently checks token validity, reachability, and
|
||||||
|
manifest↔disk integrity — but not "does the provider's response still look
|
||||||
|
like what the parser expects."
|
||||||
|
|
||||||
|
To investigate: a lightweight schema/shape assertion on a known-good sample
|
||||||
|
of each provider's listing + conversation-detail responses (presence and type
|
||||||
|
of the fields the normalizers rely on), surfaced as a `doctor` check or a
|
||||||
|
dedicated canary.
|
||||||
|
|
||||||
|
**Live recon — Claude captured 2026-06-28 (ChatGPT pending a token refresh):**
|
||||||
|
|
||||||
|
- **Dependency surface (assert ONLY these — see below for why):**
|
||||||
|
- listing item: `uuid`, `name`, `updated_at`/`created_at`, `project.name`.
|
||||||
|
- conversation detail: `uuid`/`id`, `name`, `created_at`, `updated_at`,
|
||||||
|
`project.name`, `chat_messages[]`.
|
||||||
|
- message: `sender` (`human`/`assistant`), `text` (string) or `content`
|
||||||
|
(list of typed blocks), `created_at`.
|
||||||
|
- **Key finding — full-shape diffing is the wrong design.** Claude's
|
||||||
|
`settings` object is full of volatile internal codenames that churn
|
||||||
|
constantly: `enabled_bananagrams`, `enabled_sourdough`, `enabled_foccacia`,
|
||||||
|
`enabled_saffron`, `enabled_turmeric`, `enabled_monkeys_in_a_barrel`,
|
||||||
|
`paprika_mode`, `enabled_megaminds`, … A "any new/removed key = drift"
|
||||||
|
canary would fire on every UI experiment. The canary MUST target the
|
||||||
|
normalizer's load-bearing fields only, not the whole response. (Aligns with
|
||||||
|
the drop-noise-don't-retain-it principle.)
|
||||||
|
- **Real Claude messages are flat `text`/`sender`** — in this archive every
|
||||||
|
message had a string `text` and NO `content` block list (0 rich blocks
|
||||||
|
observed). So `_extract_claude_blocks` / `_dispatch_claude_block` (tool_use,
|
||||||
|
thinking, image, …) is an **unexercised theoretical path**; drift there
|
||||||
|
can't be "caught" by a canary because it never runs on real data — it's a
|
||||||
|
safety net for if Claude ever switches to block content. The canary should
|
||||||
|
assert the flat shape and *warn if `content` ever appears as a list* (that
|
||||||
|
itself is the drift event that would activate the dormant code).
|
||||||
|
- **Possible silent-loss spot (separate from drift):** Claude messages carry
|
||||||
|
`attachments` and `files` arrays (empty in this sample) that the normalizer
|
||||||
|
ignores entirely. If a user ever attaches files in Claude, they'd be
|
||||||
|
dropped without a LossReport entry. Worth a follow-up check.
|
||||||
|
**Live recon — ChatGPT captured 2026-06-28:**
|
||||||
|
|
||||||
|
- **Dependency surface (assert ONLY these):**
|
||||||
|
- listing item: `id`, `title`, `update_time`/`create_time`.
|
||||||
|
- conversation detail: `conversation_id`/`id`, `title`, `create_time`,
|
||||||
|
`update_time`, `mapping` (non-empty).
|
||||||
|
- mapping node: `message`, `children` (the tree walk depends on both);
|
||||||
|
message: `author.role`, `author.name`, `content.content_type`,
|
||||||
|
`content.parts`, `metadata.is_visually_hidden_from_conversation`.
|
||||||
|
- **content_type vocabulary observed (all currently handled):** `text`,
|
||||||
|
`model_editable_context`, `multimodal_text`, `thoughts`, `code`,
|
||||||
|
`execution_output`, `reasoning_recap`, `user_editable_context`,
|
||||||
|
`tether_browsing_display`. A *new* content_type already degrades gracefully
|
||||||
|
(visible `unknown` block + WARNING + LossReport tally) — so content_type
|
||||||
|
drift is **already non-silent**. The canary just needs to confirm the known
|
||||||
|
set still parses to non-empty blocks.
|
||||||
|
- **The genuinely silent drift risk — `author.name` collapse keys.**
|
||||||
|
`_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}`. Recon
|
||||||
|
confirms `file_search` is live (and `web.run`/`python` are correctly left
|
||||||
|
un-collapsed). If OpenAI renames `file_search`, the collapse **silently
|
||||||
|
stops** and the archive re-bloats with no error or LossReport entry. This is
|
||||||
|
the top canary target: assert that retrieval-dump tool authors are still
|
||||||
|
recognized, or at least flag unfamiliar `(role="tool", author.name)` pairs.
|
||||||
|
- **Second silent risk — empty `content.parts`.** A `text` message whose
|
||||||
|
`parts` field is renamed/emptied yields zero blocks and is skipped with only
|
||||||
|
a debug log = silent loss. Canary should assert a sampled `text` message
|
||||||
|
produces a non-empty block.
|
||||||
|
|
||||||
|
**Recon complete for both providers. Canary design (ready to build):** a
|
||||||
|
`doctor` check (or dedicated `canary` command) that, per provider, fetches one
|
||||||
|
listing page + one conversation and asserts the dependency-surface fields
|
||||||
|
above by presence+type — NOT full shape (Claude `settings` codenames prove
|
||||||
|
full-shape diffing is pure noise). Specific tripwires: (ChatGPT) unfamiliar
|
||||||
|
`(tool, author.name)` pair and empty `parts` on a text message; (Claude)
|
||||||
|
`content` appearing as a list, and non-empty `attachments`/`files`. Failures
|
||||||
|
surface as a warning, never a hard error (a backup tool must still run).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Deprioritized
|
# Deprioritized — CLOSED as not needed (2026-06-28)
|
||||||
|
|
||||||
Kept for reference; not on the active roadmap.
|
These were considered and intentionally **not** built. Closed, not planned —
|
||||||
|
the tool is feature-complete for its purpose. Kept for reference in case a
|
||||||
|
real need ever revives one: Joplin `--force`, per-conversation cache reset,
|
||||||
|
official export-ZIP fallback, o1/o3 reasoning reclassification, Obsidian
|
||||||
|
output, token-expiry notifications (also moot — see §9), and a search
|
||||||
|
command. (Also closed, outside this list: handling Claude `attachments`/
|
||||||
|
`files`, which the canary will flag if they ever appear in real data.)
|
||||||
|
|
||||||
## Export `--force` Flag
|
## Export `--force` Flag — SHIPPED v0.6.0
|
||||||
|
|
||||||
Add `--force` to the `export` command to re-export already-cached conversations
|
Implemented 2026-06-12: `export --force` passes `force=True` to
|
||||||
without permanently clearing the entire manifest. Useful for re-generating files
|
`cache.get_new_or_updated()`. Shipped alongside a `mark_exported` fix that
|
||||||
after changing the Markdown template or output structure.
|
preserves Joplin links across re-exports, so a forced re-render + `joplin`
|
||||||
|
updates existing notes instead of duplicating them.
|
||||||
Implementation: pass a `force=True` flag to `cache.get_new_or_updated()`, which
|
|
||||||
returns all conversations regardless of cache state when force is True.
|
|
||||||
|
|
||||||
Current workaround: `python -m src.main cache --clear` then re-run export.
|
|
||||||
|
|
||||||
## Joplin `--force` Flag
|
## Joplin `--force` Flag
|
||||||
|
|
||||||
@@ -373,16 +535,13 @@ structure matching the user's Obsidian setup. No API needed — just file I/O.
|
|||||||
|
|
||||||
## Token Expiry Notifications
|
## Token Expiry Notifications
|
||||||
|
|
||||||
Proactively warn when a token is close to expiry (within 48h for ChatGPT),
|
Moved to the active roadmap as §9 (Token Validity on `doctor`). The original
|
||||||
rather than only surfacing the warning at startup. Options:
|
"proactively notify before expiry" idea is blocked by the same finding: a
|
||||||
|
reliable expiry time isn't available client-side (ChatGPT token is encrypted,
|
||||||
- Add an `expiry` subcommand that prints token status and exits non-zero if
|
Claude's is opaque, and `/api/auth/session`'s `expires` looks like a rolling
|
||||||
any token is expired or expiring soon (useful in scripts/cron)
|
window rather than the real refresh cadence). Any heads-up — a `doctor`
|
||||||
- Send a desktop notification via `notify-send` (Linux) or `osascript` (macOS)
|
line, an `expiry` subcommand, or a `notify-send` nudge — depends on first
|
||||||
when a token is within 24h of expiry
|
resolving the §9 open question of what signal is actually trustworthy.
|
||||||
|
|
||||||
Largely superseded by Brave cookie auto-extraction (#2): once refresh is
|
|
||||||
one command (or automatic), expiry warnings matter much less.
|
|
||||||
|
|
||||||
## Search Command
|
## Search Command
|
||||||
|
|
||||||
|
|||||||
@@ -79,7 +79,17 @@ Before anything else, validate your setup:
|
|||||||
ai-chat-exporter doctor
|
ai-chat-exporter doctor
|
||||||
```
|
```
|
||||||
|
|
||||||
This checks token presence, format, expiry, directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.
|
This checks token presence, format, token health (via the `/api/auth/session` `error` field), directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.
|
||||||
|
|
||||||
|
### Checking for API drift
|
||||||
|
|
||||||
|
The exporter reads ChatGPT's and Claude's undocumented internal web APIs, which can change shape without notice. The worst failure for a backup tool is *silent* — a response change that makes the exporter skip or mis-parse content without erroring. Run the canary to catch that early:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ai-chat-exporter canary
|
||||||
|
```
|
||||||
|
|
||||||
|
It probes one conversation per provider and checks only the fields the parser depends on (a renamed retrieval-tool author that would bypass the content-collapse, a new content type, drifted message fields, ignored attachments). `OK` means the live schema still matches; `WARN`/`ERROR` tells you exactly what changed. Worth running before a large export or on a schedule.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -87,23 +97,22 @@ This checks token presence, format, expiry, directory permissions, disk space, a
|
|||||||
|
|
||||||
Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
|
Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
|
||||||
|
|
||||||
### The fast way: extract from your browser
|
Tokens are entered manually — copy them from your browser's DevTools and run the wizard:
|
||||||
|
|
||||||
If the browser you're logged in with runs on the same machine as this tool:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ai-chat-exporter auth --from-browser # Brave (default)
|
ai-chat-exporter auth
|
||||||
ai-chat-exporter auth --from-browser firefox # or chrome, chromium, edge
|
|
||||||
```
|
```
|
||||||
|
|
||||||
This reads the session cookies straight from the browser's cookie store (decrypted via the OS keyring/DPAPI/Keychain), validates each token against the live API, and writes `.env` — no DevTools, no copy-pasting. A token that fails validation never overwrites a working one. If extraction fails (browser not installed here, cookie DB locked, logged out), fall back to the manual flow below.
|
The wizard detects your OS, shows the correct DevTools shortcut, and writes the values to `.env` without echoing them to the terminal.
|
||||||
|
|
||||||
|
> **Why no auto-extraction from the browser?** Modern Chromium browsers (Chrome 127+, current Brave) encrypt cookies on Windows with **App-Bound Encryption**: the key is bound to the browser through a SYSTEM-level service, so reading cookies off disk requires Administrator rights and a PsExec-style SYSTEM impersonation that antivirus flags as credential theft — and it still fails on Brave specifically. There is no reliable, non-invasive way to pull these cookies automatically, so the manual DevTools flow below is the supported path.
|
||||||
|
|
||||||
### Token Lifetimes
|
### Token Lifetimes
|
||||||
|
|
||||||
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
||||||
|----------|-------------|----------|-----------------|
|
|----------|-------------|----------|-----------------|
|
||||||
| ChatGPT | `__Secure-next-auth.session-token.0` + `.1` | ~7 days | JWT `exp` claim (decoded automatically) |
|
| ChatGPT | `__Secure-next-auth.session-token.0` + `.1` | refresh ~weekly | `error` field of `/api/auth/session` — `doctor` reports "ChatGPT token active". The token is an encrypted JWE, so its `exp` is **not** readable client-side, and the `expires` field is a misleading rolling window; the `error` (`RefreshAccessTokenError` when dead) is the honest signal. |
|
||||||
| Claude | `sessionKey` | ~30 days | Only detectable via 401 response |
|
| Claude | `sessionKey` | ~30 days | Opaque token — only detectable via 401 response |
|
||||||
|
|
||||||
### Finding Tokens in Chrome DevTools
|
### Finding Tokens in Chrome DevTools
|
||||||
|
|
||||||
@@ -209,14 +218,26 @@ The `auth` wizard can also guide you through this step interactively.
|
|||||||
|
|
||||||
## Claude Code Sessions
|
## Claude Code Sessions
|
||||||
|
|
||||||
The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/` (override with `CLAUDE_CODE_DIR` in `.env`).
|
The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ai-chat-exporter export --provider claude-code
|
ai-chat-exporter export --provider claude-code
|
||||||
ai-chat-exporter joplin --provider claude-code
|
ai-chat-exporter joplin --provider claude-code
|
||||||
```
|
```
|
||||||
|
|
||||||
Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Each coding project becomes a Joplin notebook under the `AI-Claude` parent. The provider is included in `--provider all` whenever the sessions directory exists.
|
Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Sessions become notes under their own top-level **`AI-ClaudeCode`** Joplin notebook, in a sub-notebook per launch folder. The provider is included in `--provider all` whenever a sessions directory exists.
|
||||||
|
|
||||||
|
**Subagents.** Claude Code stores subagent (Task-tool) transcripts as separate files under `<session>/subagents/`; each is folded into its parent session inline, as a collapsible `<details>` block labeled with the subagent's type and description (its own tool traffic is collapsed like the main dialogue).
|
||||||
|
|
||||||
|
**Repo tags.** Sessions launched from a workspace root all share one folder-named notebook, so titles carry the repos each session touched — `Resume StartWRT project work [start-technologies]` — for at-a-glance scanning and search. A file's repo is the git repository it lives in (nearest ancestor with a `.git`), resolved from the tool paths in the transcript, so work is tagged wherever it happened — even across workspaces — and config/one-off files are ignored (they aren't repos). Repos are frequency-ordered and capped at 3. To never tag specific repos, set `CLAUDE_CODE_REPO_TAG_IGNORE` (comma-separated). Note this reads your current git layout, so a repo you later delete or move drops from the tag on re-export.
|
||||||
|
|
||||||
|
**Multiple locations.** By default the provider scans `~/.claude/projects/` plus `$CLAUDE_CONFIG_DIR/projects` when `CLAUDE_CONFIG_DIR` is set. To scan additional roots (e.g. other machines' sessions copied onto this box), set `CLAUDE_CODE_DIR` to a `:`-separated list:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
CLAUDE_CODE_DIR="$HOME/.claude/projects:/mnt/backup/laptop/.claude/projects"
|
||||||
|
```
|
||||||
|
|
||||||
|
Sessions from all roots are merged by folder (no per-machine label); if the same session UUID appears in two roots, the newer copy wins. Note the exporter only sees this machine's disk and cannot recover sessions Claude Code has already pruned — run it regularly.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -285,10 +306,9 @@ Each provider+project combination maps to a flat Joplin notebook created automat
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
ai-chat-exporter auth # manual wizard (DevTools flow)
|
ai-chat-exporter auth # manual wizard (DevTools flow)
|
||||||
ai-chat-exporter auth --from-browser # extract from Brave's cookie store
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut. `--from-browser [brave|chrome|chromium|edge|firefox]` skips the wizard and pulls tokens directly from a local browser, validating them live before writing `.env`.
|
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut, then writes the values to `.env`.
|
||||||
|
|
||||||
### `doctor` — Health check
|
### `doctor` — Health check
|
||||||
|
|
||||||
@@ -329,7 +349,22 @@ ai-chat-exporter export --output /path/to/my/notes
|
|||||||
ai-chat-exporter export --dry-run
|
ai-chat-exporter export --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
Options: `--provider [chatgpt|claude|claude-code|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--dry-run`
|
Options: `--provider [chatgpt|claude|claude-code|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--force`, `--dry-run`
|
||||||
|
|
||||||
|
**Re-rendering the whole archive after an upgrade.** New formatting or features (collapse policy, media downloads) only change conversations as they're re-exported. To re-render everything you already have, use `--force` — it re-exports every conversation even if unchanged, **without** `cache --clear`, so your Joplin note links are preserved (a later `joplin` run updates the existing notes instead of duplicating them).
|
||||||
|
|
||||||
|
A force re-render runs as a tracked "campaign": each run re-renders the least-recently-exported conversations, the "still to go" count shrinks each run, and finished providers do no further work. The simplest approach is one uncapped pass:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ai-chat-exporter export --force # re-renders everything in one paced pass
|
||||||
|
ai-chat-exporter joplin # update notes + upload media
|
||||||
|
```
|
||||||
|
|
||||||
|
Or spread the load with `--max-conversations`, re-running until it reports "Force re-render complete":
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ai-chat-exporter export --force --max-conversations 50 # repeat until complete
|
||||||
|
```
|
||||||
|
|
||||||
### `list` — List conversations
|
### `list` — List conversations
|
||||||
|
|
||||||
|
|||||||
+1
-2
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "ai-chat-exporter"
|
name = "ai-chat-exporter"
|
||||||
version = "0.6.0"
|
version = "0.8.0"
|
||||||
description = "Export ChatGPT and Claude conversation history to Markdown for personal archival in Joplin"
|
description = "Export ChatGPT and Claude conversation history to Markdown for personal archival in Joplin"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
@@ -15,7 +15,6 @@ dependencies = [
|
|||||||
"rich==13.7.1",
|
"rich==13.7.1",
|
||||||
"python-slugify==8.0.4",
|
"python-slugify==8.0.4",
|
||||||
"PyJWT==2.8.0",
|
"PyJWT==2.8.0",
|
||||||
"browser-cookie3==0.20.1",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[project.optional-dependencies]
|
[project.optional-dependencies]
|
||||||
|
|||||||
@@ -26,4 +26,3 @@ setuptools==82.0.0
|
|||||||
text-unidecode==1.3
|
text-unidecode==1.3
|
||||||
urllib3==2.6.3
|
urllib3==2.6.3
|
||||||
wheel==0.46.3
|
wheel==0.46.3
|
||||||
browser-cookie3==0.20.1
|
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ BLOCK_TYPE_FILE_PLACEHOLDER = "file_placeholder"
|
|||||||
BLOCK_TYPE_UNKNOWN = "unknown"
|
BLOCK_TYPE_UNKNOWN = "unknown"
|
||||||
BLOCK_TYPE_HIDDEN_CONTEXT_MARKER = "hidden_context_marker"
|
BLOCK_TYPE_HIDDEN_CONTEXT_MARKER = "hidden_context_marker"
|
||||||
BLOCK_TYPE_COLLAPSED = "collapsed"
|
BLOCK_TYPE_COLLAPSED = "collapsed"
|
||||||
|
BLOCK_TYPE_SUBAGENT = "subagent"
|
||||||
|
|
||||||
COLLAPSED_KIND_TOOL_DUMP = "tool_dump"
|
COLLAPSED_KIND_TOOL_DUMP = "tool_dump"
|
||||||
COLLAPSED_KIND_HIDDEN_CONTEXT = "hidden_context"
|
COLLAPSED_KIND_HIDDEN_CONTEXT = "hidden_context"
|
||||||
@@ -36,6 +37,13 @@ UNKNOWN_REASON_UNKNOWN_FIELD_IN_KNOWN_TYPE = "unknown_field_in_known_type"
|
|||||||
|
|
||||||
_OBSERVED_KEYS_LIMIT = 10
|
_OBSERVED_KEYS_LIMIT = 10
|
||||||
|
|
||||||
|
# Role labels for the turns rendered inside a folded subagent <details> block.
|
||||||
|
_SUBAGENT_ROLE_LABELS = {
|
||||||
|
"user": "🧑 Human",
|
||||||
|
"assistant": "🤖 Assistant",
|
||||||
|
"tool": "🔧 Tool",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Constructors
|
# Constructors
|
||||||
@@ -193,6 +201,31 @@ def make_collapsed_block(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def make_subagent_block(
|
||||||
|
agent_type: str,
|
||||||
|
description: str,
|
||||||
|
messages: list[dict],
|
||||||
|
) -> dict:
|
||||||
|
"""A folded Claude Code subagent (Task-tool) transcript.
|
||||||
|
|
||||||
|
Claude Code stores subagent transcripts as separate ``subagents/*.jsonl``
|
||||||
|
files; the provider folds each one into its parent session at the point the
|
||||||
|
``Task``/``Agent`` tool spawned it. ``messages`` is a normal list of
|
||||||
|
``{role, blocks}`` dicts (the same shape as a conversation's messages),
|
||||||
|
extracted under the same hidden-content policy as the main dialogue — so the
|
||||||
|
subagent's own tool traffic is already collapsed inside these blocks.
|
||||||
|
|
||||||
|
Rendered as a collapsible ``<details>`` element so the main conversation
|
||||||
|
stays readable while the deliverable is preserved (and stays searchable).
|
||||||
|
"""
|
||||||
|
return {
|
||||||
|
"type": BLOCK_TYPE_SUBAGENT,
|
||||||
|
"agent_type": agent_type or "",
|
||||||
|
"description": description or "",
|
||||||
|
"messages": messages or [],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def make_hidden_context_marker(content_type: str) -> dict:
|
def make_hidden_context_marker(content_type: str) -> dict:
|
||||||
"""A short prepend block that flags the surrounding message as hidden context.
|
"""A short prepend block that flags the surrounding message as hidden context.
|
||||||
|
|
||||||
@@ -322,6 +355,25 @@ def _render_one(block: dict) -> str:
|
|||||||
keys_str = ", ".join(f"`{k}`" for k in keys)
|
keys_str = ", ".join(f"`{k}`" for k in keys)
|
||||||
lines.append(f"Keys observed: {keys_str}")
|
lines.append(f"Keys observed: {keys_str}")
|
||||||
return _blockquote_prefix("\n".join(lines))
|
return _blockquote_prefix("\n".join(lines))
|
||||||
|
if btype == BLOCK_TYPE_SUBAGENT:
|
||||||
|
agent_type = block.get("agent_type") or "subagent"
|
||||||
|
description = block.get("description") or ""
|
||||||
|
summary = f"🤖 Subagent: {agent_type}"
|
||||||
|
if description:
|
||||||
|
summary += f" — {description}"
|
||||||
|
parts = ["<details>", f"<summary>{summary}</summary>", ""]
|
||||||
|
for msg in block.get("messages") or []:
|
||||||
|
body = render_blocks_to_markdown(msg.get("blocks") or [])
|
||||||
|
if not body.strip():
|
||||||
|
continue
|
||||||
|
role = msg.get("role", "assistant")
|
||||||
|
label = _SUBAGENT_ROLE_LABELS.get(role, f"💬 {role.capitalize()}")
|
||||||
|
parts.append(f"**{label}**")
|
||||||
|
parts.append("")
|
||||||
|
parts.append(body)
|
||||||
|
parts.append("")
|
||||||
|
parts.append("</details>")
|
||||||
|
return "\n".join(parts)
|
||||||
if btype == BLOCK_TYPE_HIDDEN_CONTEXT_MARKER:
|
if btype == BLOCK_TYPE_HIDDEN_CONTEXT_MARKER:
|
||||||
ctype = block.get("content_type", "")
|
ctype = block.get("content_type", "")
|
||||||
return f"> ℹ️ **Hidden context** — `{ctype}`"
|
return f"> ℹ️ **Hidden context** — `{ctype}`"
|
||||||
|
|||||||
@@ -1,101 +0,0 @@
|
|||||||
"""Extract session tokens directly from a local browser's cookie store.
|
|
||||||
|
|
||||||
Default browser is Brave. Uses browser-cookie3, which handles the
|
|
||||||
platform-specific cookie decryption (Linux: AES key derived from the OS
|
|
||||||
keyring "Safe Storage" secret; Windows: DPAPI; macOS: Keychain). The browser
|
|
||||||
must be the one the user is actually logged in with, on the same machine.
|
|
||||||
|
|
||||||
Failure modes worth knowing:
|
|
||||||
- Browser not installed / profile not found → BrowserTokenError
|
|
||||||
- Cookie DB locked (browser running with exclusive lock; rare on Linux,
|
|
||||||
common on Windows) → BrowserTokenError suggesting the browser be closed
|
|
||||||
- Logged out / cookie expired → BrowserTokenError naming the missing cookie
|
|
||||||
"""
|
|
||||||
|
|
||||||
import logging
|
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
DEFAULT_BROWSER = "brave"
|
|
||||||
SUPPORTED_BROWSERS = ("brave", "chrome", "chromium", "edge", "firefox")
|
|
||||||
|
|
||||||
# ChatGPT splits large session tokens across two cookies to stay under the
|
|
||||||
# browser's 4KB cookie limit; small tokens use the unsuffixed name.
|
|
||||||
_CHATGPT_COOKIE_0 = "__Secure-next-auth.session-token.0"
|
|
||||||
_CHATGPT_COOKIE_1 = "__Secure-next-auth.session-token.1"
|
|
||||||
_CHATGPT_COOKIE_SINGLE = "__Secure-next-auth.session-token"
|
|
||||||
_CLAUDE_COOKIE = "sessionKey"
|
|
||||||
|
|
||||||
|
|
||||||
class BrowserTokenError(Exception):
|
|
||||||
"""Raised when tokens cannot be extracted from the browser profile."""
|
|
||||||
|
|
||||||
|
|
||||||
def _load_cookies(browser: str, domain: str) -> dict[str, str]:
|
|
||||||
"""Return {cookie_name: value} for ``domain`` from the given browser."""
|
|
||||||
if browser not in SUPPORTED_BROWSERS:
|
|
||||||
raise BrowserTokenError(
|
|
||||||
f"Unsupported browser {browser!r}. "
|
|
||||||
f"Supported: {', '.join(SUPPORTED_BROWSERS)}"
|
|
||||||
)
|
|
||||||
|
|
||||||
import browser_cookie3
|
|
||||||
|
|
||||||
loader = getattr(browser_cookie3, browser)
|
|
||||||
try:
|
|
||||||
jar = loader(domain_name=domain)
|
|
||||||
except Exception as e:
|
|
||||||
# browser_cookie3 raises BrowserCookieError plus assorted OS/keyring
|
|
||||||
# errors. All mean the same thing to the caller: fall back to manual.
|
|
||||||
hint = ""
|
|
||||||
text = str(e).lower()
|
|
||||||
if "lock" in text or "database is locked" in text:
|
|
||||||
hint = " Close the browser and try again."
|
|
||||||
elif "not find" in text or "no such file" in text:
|
|
||||||
hint = " Is the browser installed on this machine?"
|
|
||||||
raise BrowserTokenError(
|
|
||||||
f"Could not read {browser} cookies for {domain}: {e}.{hint}"
|
|
||||||
) from e
|
|
||||||
|
|
||||||
return {cookie.name: cookie.value or "" for cookie in jar}
|
|
||||||
|
|
||||||
|
|
||||||
def extract_chatgpt_tokens(browser: str = DEFAULT_BROWSER) -> tuple[str, str | None]:
|
|
||||||
"""Return (session_token_0, session_token_1_or_None) from the browser.
|
|
||||||
|
|
||||||
Raises BrowserTokenError when no session cookie is present (not logged in,
|
|
||||||
or logged out since the last visit).
|
|
||||||
"""
|
|
||||||
cookies = _load_cookies(browser, "chatgpt.com")
|
|
||||||
token_0 = cookies.get(_CHATGPT_COOKIE_0, "").strip()
|
|
||||||
token_1 = cookies.get(_CHATGPT_COOKIE_1, "").strip() or None
|
|
||||||
if token_0:
|
|
||||||
logger.info(
|
|
||||||
"[browser-tokens] ChatGPT session cookies found in %s (chunked: %s)",
|
|
||||||
browser,
|
|
||||||
"yes" if token_1 else "no",
|
|
||||||
)
|
|
||||||
return token_0, token_1
|
|
||||||
|
|
||||||
single = cookies.get(_CHATGPT_COOKIE_SINGLE, "").strip()
|
|
||||||
if single:
|
|
||||||
logger.info("[browser-tokens] ChatGPT session cookie found in %s (single)", browser)
|
|
||||||
return single, None
|
|
||||||
|
|
||||||
raise BrowserTokenError(
|
|
||||||
f"No ChatGPT session cookie in {browser} — open https://chatgpt.com "
|
|
||||||
"in that browser and log in, then retry."
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def extract_claude_session(browser: str = DEFAULT_BROWSER) -> str:
|
|
||||||
"""Return the Claude ``sessionKey`` cookie value from the browser."""
|
|
||||||
cookies = _load_cookies(browser, "claude.ai")
|
|
||||||
session_key = cookies.get(_CLAUDE_COOKIE, "").strip()
|
|
||||||
if session_key:
|
|
||||||
logger.info("[browser-tokens] Claude sessionKey found in %s", browser)
|
|
||||||
return session_key
|
|
||||||
raise BrowserTokenError(
|
|
||||||
f"No Claude sessionKey cookie in {browser} — open https://claude.ai "
|
|
||||||
"in that browser and log in, then retry."
|
|
||||||
)
|
|
||||||
+64
-3
@@ -84,7 +84,13 @@ class Cache:
|
|||||||
if provider not in self._data:
|
if provider not in self._data:
|
||||||
self._data[provider] = {}
|
self._data[provider] = {}
|
||||||
|
|
||||||
self._data[provider][conv_id] = {
|
# Preserve the Joplin link across re-exports. Without this, re-rendering
|
||||||
|
# a conversation (e.g. a forced re-export to pick up new formatting)
|
||||||
|
# would drop joplin_note_id, and the next sync would create a duplicate
|
||||||
|
# note instead of updating the existing one. exported_at is refreshed,
|
||||||
|
# so get_joplin_pending() still flags the note for an update.
|
||||||
|
prev = self._data[provider].get(conv_id) or {}
|
||||||
|
entry = {
|
||||||
"title": metadata.get("title", ""),
|
"title": metadata.get("title", ""),
|
||||||
"project": metadata.get("project"),
|
"project": metadata.get("project"),
|
||||||
"created_at": metadata.get("created_at", ""),
|
"created_at": metadata.get("created_at", ""),
|
||||||
@@ -92,20 +98,56 @@ class Cache:
|
|||||||
"exported_at": datetime.now(tz=timezone.utc).isoformat(),
|
"exported_at": datetime.now(tz=timezone.utc).isoformat(),
|
||||||
"file_path": metadata.get("file_path", ""),
|
"file_path": metadata.get("file_path", ""),
|
||||||
}
|
}
|
||||||
|
for carried in ("joplin_note_id", "joplin_synced_at", "joplin_resources"):
|
||||||
|
if carried in prev:
|
||||||
|
entry[carried] = prev[carried]
|
||||||
|
|
||||||
|
self._data[provider][conv_id] = entry
|
||||||
self._data["last_run"] = datetime.now(tz=timezone.utc).isoformat()
|
self._data["last_run"] = datetime.now(tz=timezone.utc).isoformat()
|
||||||
self._save()
|
self._save()
|
||||||
|
|
||||||
def get_new_or_updated(self, provider: str, conversations: list[dict]) -> list[dict]:
|
def get_new_or_updated(
|
||||||
|
self,
|
||||||
|
provider: str,
|
||||||
|
conversations: list[dict],
|
||||||
|
force: bool = False,
|
||||||
|
campaign_at: str | None = None,
|
||||||
|
) -> list[dict]:
|
||||||
"""Filter a conversation list to only new or updated conversations.
|
"""Filter a conversation list to only new or updated conversations.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
provider: "chatgpt" or "claude"
|
provider: "chatgpt" or "claude"
|
||||||
conversations: List of raw conversation dicts from the provider.
|
conversations: List of raw conversation dicts from the provider.
|
||||||
Each must have an ``id``/``uuid`` and ``updated_at``/``update_time``.
|
Each must have an ``id``/``uuid`` and ``updated_at``/``update_time``.
|
||||||
|
force: When True, return conversations to re-render regardless of
|
||||||
|
cache freshness — used by ``export --force``.
|
||||||
|
campaign_at: In force mode, exclude conversations already re-exported
|
||||||
|
at/after this ISO timestamp (the re-render campaign start), so
|
||||||
|
each run advances and finished providers do nothing.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Subset that needs to be exported.
|
Subset that needs to be exported, ordered oldest-export-first in
|
||||||
|
force mode.
|
||||||
"""
|
"""
|
||||||
|
if force:
|
||||||
|
# Re-export for a re-render campaign, ordered least-recently-exported
|
||||||
|
# first (never-exported sorts first). When ``campaign_at`` is given,
|
||||||
|
# conversations already re-exported during this campaign
|
||||||
|
# (exported_at >= campaign_at) are excluded — so finished providers
|
||||||
|
# do no work and the remaining count shrinks to zero. Each capped run
|
||||||
|
# re-renders the next-oldest batch. Conversations without an id drop.
|
||||||
|
entries = self._data.get(provider, {})
|
||||||
|
|
||||||
|
def _exported_at(conv: dict) -> str:
|
||||||
|
conv_id = conv.get("id") or conv.get("uuid", "")
|
||||||
|
entry = entries.get(conv_id) or {}
|
||||||
|
return entry.get("exported_at") or ""
|
||||||
|
|
||||||
|
candidates = [c for c in conversations if c.get("id") or c.get("uuid")]
|
||||||
|
if campaign_at is not None:
|
||||||
|
candidates = [c for c in candidates if _exported_at(c) < campaign_at]
|
||||||
|
return sorted(candidates, key=_exported_at)
|
||||||
|
|
||||||
result = []
|
result = []
|
||||||
for conv in conversations:
|
for conv in conversations:
|
||||||
conv_id = conv.get("id") or conv.get("uuid", "")
|
conv_id = conv.get("id") or conv.get("uuid", "")
|
||||||
@@ -249,6 +291,25 @@ class Cache:
|
|||||||
"""Return the ISO8601 timestamp of the last export run, or None."""
|
"""Return the ISO8601 timestamp of the last export run, or None."""
|
||||||
return self._data.get("last_run")
|
return self._data.get("last_run")
|
||||||
|
|
||||||
|
def exported_at(self, provider: str, conv_id: str) -> str:
|
||||||
|
"""Return the recorded ``exported_at`` for a conversation, or '' if absent."""
|
||||||
|
entry = self._data.get(provider, {}).get(conv_id)
|
||||||
|
return entry.get("exported_at", "") if isinstance(entry, dict) else ""
|
||||||
|
|
||||||
|
# Force re-render campaign marker (top-level scalar; skipped by stats/clear,
|
||||||
|
# which only act on dict-valued provider keys).
|
||||||
|
def get_force_campaign(self) -> str | None:
|
||||||
|
"""Return the active force-re-render campaign start timestamp, or None."""
|
||||||
|
return self._data.get("force_campaign_at")
|
||||||
|
|
||||||
|
def set_force_campaign(self, started_at: str) -> None:
|
||||||
|
self._data["force_campaign_at"] = started_at
|
||||||
|
self._save()
|
||||||
|
|
||||||
|
def clear_force_campaign(self) -> None:
|
||||||
|
if self._data.pop("force_campaign_at", None) is not None:
|
||||||
|
self._save()
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Private helpers
|
# Private helpers
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|||||||
+16
-5
@@ -178,19 +178,28 @@ class JoplinClient:
|
|||||||
logger.info("[joplin] Note created: %r → %s", title, note_id)
|
logger.info("[joplin] Note created: %r → %s", title, note_id)
|
||||||
return note_id
|
return note_id
|
||||||
|
|
||||||
def update_note(self, note_id: str, title: str, body: str) -> None:
|
def update_note(
|
||||||
|
self, note_id: str, title: str, body: str, parent_id: str | None = None
|
||||||
|
) -> None:
|
||||||
"""Update the title and body of an existing note.
|
"""Update the title and body of an existing note.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
note_id: Joplin note ID.
|
note_id: Joplin note ID.
|
||||||
title: New note title.
|
title: New note title.
|
||||||
body: New note body (Markdown).
|
body: New note body (Markdown).
|
||||||
|
parent_id: If given, also move the note into this notebook. Joplin's
|
||||||
|
``PUT /notes/:id`` relocates a note when ``parent_id`` is set —
|
||||||
|
this is how notes self-heal to a new notebook mapping (e.g. the
|
||||||
|
Claude Code AI-Claude → AI-ClaudeCode migration) on re-sync.
|
||||||
"""
|
"""
|
||||||
logger.debug(
|
logger.debug(
|
||||||
"[joplin] Updating note %s: %r (%d chars)",
|
"[joplin] Updating note %s: %r (%d chars)",
|
||||||
note_id, title, len(body),
|
note_id, title, len(body),
|
||||||
)
|
)
|
||||||
self._put(f"/notes/{note_id}", {"title": title, "body": body})
|
data: dict = {"title": title, "body": body}
|
||||||
|
if parent_id:
|
||||||
|
data["parent_id"] = parent_id
|
||||||
|
self._put(f"/notes/{note_id}", data)
|
||||||
logger.info("[joplin] Note updated: %r (%s)", title, note_id)
|
logger.info("[joplin] Note updated: %r (%s)", title, note_id)
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -395,9 +404,11 @@ def upload_media_and_rewrite(
|
|||||||
_PROVIDER_DISPLAY = {
|
_PROVIDER_DISPLAY = {
|
||||||
"chatgpt": "AI-ChatGPT",
|
"chatgpt": "AI-ChatGPT",
|
||||||
"claude": "AI-Claude",
|
"claude": "AI-Claude",
|
||||||
# Decision 2026-06-12: Claude Code coding projects nest under the same
|
# Decision 2026-07-06: Claude Code coding sessions get their own top-level
|
||||||
# AI-Claude parent, alongside Claude web projects.
|
# notebook (was nested under AI-Claude, which made them hard to find,
|
||||||
"claude-code": "AI-Claude",
|
# intermixed with Claude web projects and named by dev folder). Existing
|
||||||
|
# notes self-heal into here on the next sync — update_note moves them.
|
||||||
|
"claude-code": "AI-ClaudeCode",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+185
-149
@@ -7,7 +7,7 @@ import platform
|
|||||||
import shutil
|
import shutil
|
||||||
import sys
|
import sys
|
||||||
import traceback
|
import traceback
|
||||||
from datetime import datetime
|
from datetime import datetime, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import click
|
import click
|
||||||
@@ -18,7 +18,7 @@ from src.cache import Cache, CacheError
|
|||||||
from src.config import ConfigError
|
from src.config import ConfigError
|
||||||
from src.logging_config import setup_logging
|
from src.logging_config import setup_logging
|
||||||
from src.loss_report import LossReport
|
from src.loss_report import LossReport
|
||||||
from src.providers.base import ProviderError
|
from src.providers.base import DRIFT_ERROR, DRIFT_OK, DRIFT_WARN, ProviderError
|
||||||
|
|
||||||
console = Console()
|
console = Console()
|
||||||
err_console = Console(stderr=True)
|
err_console = Console(stderr=True)
|
||||||
@@ -62,6 +62,14 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, debug: bool, no_log_file
|
|||||||
"""Export ChatGPT and Claude conversations to Markdown for personal archival."""
|
"""Export ChatGPT and Claude conversations to Markdown for personal archival."""
|
||||||
ctx.ensure_object(dict)
|
ctx.ensure_object(dict)
|
||||||
|
|
||||||
|
# Load .env BEFORE reading any env var (LOG_FILE, CACHE_DIR). Without this
|
||||||
|
# the cache/log paths fell back to defaults regardless of .env — the cache
|
||||||
|
# in particular silently lived at ./cache even when CACHE_DIR was set.
|
||||||
|
# override=False so a real shell env var still wins over .env.
|
||||||
|
import os
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
load_dotenv(override=False)
|
||||||
|
|
||||||
# Determine console log level
|
# Determine console log level
|
||||||
if debug or verbose:
|
if debug or verbose:
|
||||||
level = logging.DEBUG
|
level = logging.DEBUG
|
||||||
@@ -71,7 +79,6 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, debug: bool, no_log_file
|
|||||||
level = logging.INFO
|
level = logging.INFO
|
||||||
|
|
||||||
# Determine log file path from env (setup_logging handles "none")
|
# Determine log file path from env (setup_logging handles "none")
|
||||||
import os
|
|
||||||
log_file = os.getenv("LOG_FILE", "./cache/logs/exporter.log")
|
log_file = os.getenv("LOG_FILE", "./cache/logs/exporter.log")
|
||||||
|
|
||||||
setup_logging(level=level, log_file=log_file, no_log_file=no_log_file)
|
setup_logging(level=level, log_file=log_file, no_log_file=no_log_file)
|
||||||
@@ -110,37 +117,17 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, debug: bool, no_log_file
|
|||||||
|
|
||||||
|
|
||||||
@cli.command()
|
@cli.command()
|
||||||
@click.option(
|
|
||||||
"--from-browser",
|
|
||||||
"from_browser",
|
|
||||||
is_flag=False,
|
|
||||||
flag_value="brave",
|
|
||||||
default=None,
|
|
||||||
metavar="[BROWSER]",
|
|
||||||
help=(
|
|
||||||
"Extract tokens straight from a local browser's cookie store instead "
|
|
||||||
"of the manual DevTools flow. Defaults to brave when no browser is "
|
|
||||||
"named; also supports chrome, chromium, edge, firefox. The browser "
|
|
||||||
"must be installed and logged in on this machine."
|
|
||||||
),
|
|
||||||
)
|
|
||||||
@click.pass_context
|
@click.pass_context
|
||||||
def auth(ctx: click.Context, from_browser: str | None) -> None:
|
def auth(ctx: click.Context) -> None:
|
||||||
"""Interactive setup wizard for session tokens.
|
"""Interactive setup wizard for session tokens.
|
||||||
|
|
||||||
Guides you through finding and saving your ChatGPT and Claude session
|
Guides you through finding and saving your ChatGPT and Claude session
|
||||||
tokens. Tokens are never echoed to the terminal. With --from-browser,
|
tokens. Tokens are never echoed to the terminal.
|
||||||
tokens are pulled from the browser's cookie store and written to .env
|
|
||||||
without any copy-pasting.
|
|
||||||
|
|
||||||
Token lifetimes:
|
Token lifetimes:
|
||||||
ChatGPT (__Secure-next-auth.session-token): ~7 days (JWT)
|
ChatGPT (__Secure-next-auth.session-token): ~7 days (JWT)
|
||||||
Claude (sessionKey): ~30 days (opaque string)
|
Claude (sessionKey): ~30 days (opaque string)
|
||||||
"""
|
"""
|
||||||
if from_browser:
|
|
||||||
_auth_from_browser(from_browser.lower())
|
|
||||||
return
|
|
||||||
|
|
||||||
os_name = platform.system()
|
os_name = platform.system()
|
||||||
|
|
||||||
console.print("\n[bold cyan]AI Chat Exporter — Token Setup Wizard[/bold cyan]\n")
|
console.print("\n[bold cyan]AI Chat Exporter — Token Setup Wizard[/bold cyan]\n")
|
||||||
@@ -166,8 +153,6 @@ def auth(ctx: click.Context, from_browser: str | None) -> None:
|
|||||||
|
|
||||||
|
|
||||||
def _auth_chatgpt(os_name: str) -> None:
|
def _auth_chatgpt(os_name: str) -> None:
|
||||||
import jwt as pyjwt
|
|
||||||
|
|
||||||
console.print("\n[bold]─── ChatGPT ───[/bold]")
|
console.print("\n[bold]─── ChatGPT ───[/bold]")
|
||||||
console.print("1. Open [link=https://chatgpt.com]https://chatgpt.com[/link] and log in.")
|
console.print("1. Open [link=https://chatgpt.com]https://chatgpt.com[/link] and log in.")
|
||||||
if os_name == "Darwin":
|
if os_name == "Darwin":
|
||||||
@@ -188,22 +173,12 @@ def _auth_chatgpt(os_name: str) -> None:
|
|||||||
|
|
||||||
token_1 = click.prompt("ChatGPT session token (.1, leave blank if absent)", hide_input=True, default="", show_default=False).strip() or None
|
token_1 = click.prompt("ChatGPT session token (.1, leave blank if absent)", hide_input=True, default="", show_default=False).strip() or None
|
||||||
|
|
||||||
# Validate
|
# Format hint only — the token is a JWE whose expiry can't be decoded
|
||||||
|
# client-side, so validity is judged by the live check below (the
|
||||||
|
# /api/auth/session `error` field), not by parsing the token. See §9.
|
||||||
if not token.startswith("eyJ"):
|
if not token.startswith("eyJ"):
|
||||||
console.print("[yellow]Warning: token doesn't look like a JWT (expected 'eyJ...').[/yellow]")
|
console.print("[yellow]Warning: token doesn't look like a JWT (expected 'eyJ...').[/yellow]")
|
||||||
|
|
||||||
expiry_str = ""
|
|
||||||
try:
|
|
||||||
payload = pyjwt.decode(token, options={"verify_signature": False})
|
|
||||||
exp = payload.get("exp")
|
|
||||||
if exp:
|
|
||||||
from datetime import timezone
|
|
||||||
expiry = datetime.fromtimestamp(exp, tz=timezone.utc)
|
|
||||||
expiry_str = expiry.strftime("%Y-%m-%d %H:%M UTC")
|
|
||||||
console.print(f"[green]Token decoded — expires: {expiry_str}[/green]")
|
|
||||||
except Exception:
|
|
||||||
console.print("[yellow]Could not decode token expiry.[/yellow]")
|
|
||||||
|
|
||||||
# Live validation — exchange session token for an access token
|
# Live validation — exchange session token for an access token
|
||||||
_valid = False
|
_valid = False
|
||||||
_error: str | None = None
|
_error: str | None = None
|
||||||
@@ -300,82 +275,6 @@ def _auth_claude(os_name: str) -> None:
|
|||||||
_write_token_to_env("CLAUDE_SESSION_KEY", key)
|
_write_token_to_env("CLAUDE_SESSION_KEY", key)
|
||||||
|
|
||||||
|
|
||||||
def _auth_from_browser(browser: str) -> None:
|
|
||||||
"""Extract ChatGPT + Claude tokens from a local browser and write .env.
|
|
||||||
|
|
||||||
Each provider is validated against its live API before anything is
|
|
||||||
written, so a stale cookie (logged out since last visit) never replaces
|
|
||||||
a working token. Exits 1 only when nothing could be configured.
|
|
||||||
"""
|
|
||||||
from src.browser_tokens import (
|
|
||||||
BrowserTokenError,
|
|
||||||
SUPPORTED_BROWSERS,
|
|
||||||
extract_chatgpt_tokens,
|
|
||||||
extract_claude_session,
|
|
||||||
)
|
|
||||||
|
|
||||||
if browser not in SUPPORTED_BROWSERS:
|
|
||||||
err_console.print(
|
|
||||||
f"[red]Unsupported browser '{browser}'. "
|
|
||||||
f"Supported: {', '.join(SUPPORTED_BROWSERS)}[/red]"
|
|
||||||
)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
console.print(f"\n[bold cyan]Extracting session tokens from {browser}…[/bold cyan]\n")
|
|
||||||
configured = 0
|
|
||||||
|
|
||||||
# ── ChatGPT ──────────────────────────────────────────────────────────
|
|
||||||
try:
|
|
||||||
token, token_1 = extract_chatgpt_tokens(browser)
|
|
||||||
with console.status("[dim]Validating ChatGPT token…[/dim]"):
|
|
||||||
from src.providers.chatgpt import ChatGPTProvider
|
|
||||||
prov = ChatGPTProvider(session_token=token, session_token_1=token_1)
|
|
||||||
prov._fetch_access_token()
|
|
||||||
_set_env_key("CHATGPT_SESSION_TOKEN", token)
|
|
||||||
_set_env_key("CHATGPT_SESSION_TOKEN_1", token_1 or "")
|
|
||||||
console.print("[green]ChatGPT: token extracted, validated, and saved.[/green]")
|
|
||||||
configured += 1
|
|
||||||
except BrowserTokenError as e:
|
|
||||||
console.print(f"[yellow]ChatGPT: {e}[/yellow]")
|
|
||||||
except ProviderError as e:
|
|
||||||
console.print(
|
|
||||||
f"[yellow]ChatGPT: extracted cookie failed live validation "
|
|
||||||
f"({e.original}) — .env not changed. Log in to chatgpt.com in "
|
|
||||||
f"{browser} and retry.[/yellow]"
|
|
||||||
)
|
|
||||||
|
|
||||||
# ── Claude ───────────────────────────────────────────────────────────
|
|
||||||
try:
|
|
||||||
session_key = extract_claude_session(browser)
|
|
||||||
with console.status("[dim]Validating Claude session key…[/dim]"):
|
|
||||||
from src.providers.claude import ClaudeProvider
|
|
||||||
prov = ClaudeProvider(session_key)
|
|
||||||
prov.list_conversations(offset=0, limit=1)
|
|
||||||
_set_env_key("CLAUDE_SESSION_KEY", session_key)
|
|
||||||
console.print("[green]Claude: session key extracted, validated, and saved.[/green]")
|
|
||||||
configured += 1
|
|
||||||
except BrowserTokenError as e:
|
|
||||||
console.print(f"[yellow]Claude: {e}[/yellow]")
|
|
||||||
except ProviderError as e:
|
|
||||||
console.print(
|
|
||||||
f"[yellow]Claude: extracted cookie failed live validation "
|
|
||||||
f"({e.original}) — .env not changed. Log in to claude.ai in "
|
|
||||||
f"{browser} and retry.[/yellow]"
|
|
||||||
)
|
|
||||||
|
|
||||||
if configured:
|
|
||||||
console.print(
|
|
||||||
f"\n[green]Done — {configured} provider(s) configured. "
|
|
||||||
"Run 'ai-chat-exporter doctor' to verify.[/green]"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
err_console.print(
|
|
||||||
"\n[red]No tokens could be extracted. Use the manual wizard "
|
|
||||||
"instead: ai-chat-exporter auth[/red]"
|
|
||||||
)
|
|
||||||
sys.exit(1)
|
|
||||||
|
|
||||||
|
|
||||||
def _write_token_to_env(key: str, value: str) -> None:
|
def _write_token_to_env(key: str, value: str) -> None:
|
||||||
"""Write or update a key in .env, offering to create the file if it doesn't exist."""
|
"""Write or update a key in .env, offering to create the file if it doesn't exist."""
|
||||||
if click.confirm(f"Write {key} to .env?", default=True):
|
if click.confirm(f"Write {key} to .env?", default=True):
|
||||||
@@ -433,6 +332,91 @@ def doctor(ctx: click.Context) -> None:
|
|||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
@cli.command()
|
||||||
|
@click.option(
|
||||||
|
"--provider",
|
||||||
|
type=click.Choice(["chatgpt", "claude", "all"]),
|
||||||
|
default="all",
|
||||||
|
help="Which web-API provider(s) to probe.",
|
||||||
|
)
|
||||||
|
@click.pass_context
|
||||||
|
def canary(ctx: click.Context, provider: str) -> None:
|
||||||
|
"""Probe provider APIs for schema drift against the fields the parser needs.
|
||||||
|
|
||||||
|
Fetches one listing page + one conversation per provider and checks only
|
||||||
|
the normalizer's load-bearing fields (not the full response shape, which
|
||||||
|
churns harmlessly). Surfaces the silent-failure risks for a backup tool:
|
||||||
|
a renamed retrieval-tool author bypassing the hidden-content collapse, a
|
||||||
|
new content_type, drifted message fields, or ignored attachments.
|
||||||
|
|
||||||
|
Exits non-zero if any ERROR finding is reported (a load-bearing field is
|
||||||
|
missing/mistyped). WARN findings are surfaced but do not fail. See
|
||||||
|
FUTURE.md §10.
|
||||||
|
"""
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
load_dotenv(override=False)
|
||||||
|
|
||||||
|
chatgpt_token = os.getenv("CHATGPT_SESSION_TOKEN", "").strip() or None
|
||||||
|
claude_key = os.getenv("CLAUDE_SESSION_KEY", "").strip() or None
|
||||||
|
|
||||||
|
targets: list[tuple[str, object]] = []
|
||||||
|
if provider in ("chatgpt", "all") and chatgpt_token:
|
||||||
|
from src.providers.chatgpt import ChatGPTProvider
|
||||||
|
|
||||||
|
ct1 = os.getenv("CHATGPT_SESSION_TOKEN_1", "").strip() or None
|
||||||
|
targets.append(
|
||||||
|
("chatgpt", lambda: ChatGPTProvider(chatgpt_token, session_token_1=ct1))
|
||||||
|
)
|
||||||
|
if provider in ("claude", "all") and claude_key:
|
||||||
|
from src.providers.claude import ClaudeProvider
|
||||||
|
|
||||||
|
targets.append(("claude", lambda: ClaudeProvider(claude_key)))
|
||||||
|
|
||||||
|
if not targets:
|
||||||
|
err_console.print(
|
||||||
|
"[yellow]No web-API provider tokens configured "
|
||||||
|
"(set CHATGPT_SESSION_TOKEN / CLAUDE_SESSION_KEY).[/yellow]"
|
||||||
|
)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
rows: list[tuple[str, dict]] = []
|
||||||
|
for name, factory in targets:
|
||||||
|
try:
|
||||||
|
findings = factory().check_drift()
|
||||||
|
except ProviderError as e:
|
||||||
|
findings = [{"severity": DRIFT_ERROR, "check": "probe", "detail": str(e.original)[:100]}]
|
||||||
|
except Exception as e:
|
||||||
|
findings = [{"severity": DRIFT_ERROR, "check": "probe", "detail": str(e)[:100]}]
|
||||||
|
for f in findings:
|
||||||
|
rows.append((name, f))
|
||||||
|
|
||||||
|
_print_canary_table(rows)
|
||||||
|
if any(f["severity"] == DRIFT_ERROR for _, f in rows):
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
def _print_canary_table(rows: list[tuple[str, dict]]) -> None:
|
||||||
|
table = Table(title="API Drift Canary", show_header=True)
|
||||||
|
table.add_column("Provider", style="bold")
|
||||||
|
table.add_column("Severity", justify="center")
|
||||||
|
table.add_column("Check")
|
||||||
|
table.add_column("Detail")
|
||||||
|
badge = {
|
||||||
|
DRIFT_OK: "[green]OK[/green]",
|
||||||
|
DRIFT_WARN: "[yellow]WARN[/yellow]",
|
||||||
|
DRIFT_ERROR: "[red]ERROR[/red]",
|
||||||
|
}
|
||||||
|
for name, f in rows:
|
||||||
|
table.add_row(
|
||||||
|
name,
|
||||||
|
badge.get(f["severity"], f["severity"]),
|
||||||
|
f.get("check", ""),
|
||||||
|
f.get("detail", ""),
|
||||||
|
)
|
||||||
|
console.print(table)
|
||||||
|
|
||||||
|
|
||||||
def _run_doctor_checks(cache: Cache | None = None) -> list[dict]:
|
def _run_doctor_checks(cache: Cache | None = None) -> list[dict]:
|
||||||
"""Run all doctor checks and return results.
|
"""Run all doctor checks and return results.
|
||||||
|
|
||||||
@@ -440,8 +424,6 @@ def _run_doctor_checks(cache: Cache | None = None) -> list[dict]:
|
|||||||
``file_path`` recorded in the manifest exists on disk).
|
``file_path`` recorded in the manifest exists on disk).
|
||||||
"""
|
"""
|
||||||
import os
|
import os
|
||||||
import jwt as pyjwt
|
|
||||||
from datetime import timezone
|
|
||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
# Load .env so doctor works without the user having to export vars manually
|
# Load .env so doctor works without the user having to export vars manually
|
||||||
@@ -463,25 +445,21 @@ def _run_doctor_checks(cache: Cache | None = None) -> list[dict]:
|
|||||||
if chatgpt_token:
|
if chatgpt_token:
|
||||||
is_jwt = chatgpt_token.startswith("eyJ")
|
is_jwt = chatgpt_token.startswith("eyJ")
|
||||||
add("ChatGPT token is valid JWT", is_jwt, "" if is_jwt else "Expected token starting with 'eyJ'")
|
add("ChatGPT token is valid JWT", is_jwt, "" if is_jwt else "Expected token starting with 'eyJ'")
|
||||||
|
# Expiry is NOT decodable from the token: ChatGPT's is a JWE (encrypted),
|
||||||
|
# so the `exp` claim is sealed server-side. The honest, free validity
|
||||||
|
# signal is the `error` field of /api/auth/session — `RefreshAccessTokenError`
|
||||||
|
# means the session token is dead while `expires`/`accessToken` still
|
||||||
|
# echo stale values. See FUTURE.md §9.
|
||||||
if is_jwt:
|
if is_jwt:
|
||||||
|
from src.providers.chatgpt import ChatGPTProvider
|
||||||
|
ct1 = os.getenv("CHATGPT_SESSION_TOKEN_1", "").strip() or None
|
||||||
try:
|
try:
|
||||||
payload = pyjwt.decode(chatgpt_token, options={"verify_signature": False})
|
ok, detail = ChatGPTProvider(
|
||||||
exp = payload.get("exp")
|
chatgpt_token, session_token_1=ct1
|
||||||
if exp:
|
).session_health()
|
||||||
expiry = datetime.fromtimestamp(exp, tz=timezone.utc)
|
add("ChatGPT token active", ok, detail)
|
||||||
now = datetime.now(tz=timezone.utc)
|
except Exception as e:
|
||||||
delta = expiry - now
|
add("ChatGPT token active", False, str(e)[:80])
|
||||||
detail = f"Expires {expiry.strftime('%Y-%m-%d %H:%M UTC')} ({delta.days}d)"
|
|
||||||
ok = delta.total_seconds() > 0
|
|
||||||
add("ChatGPT token not expired", ok, detail)
|
|
||||||
if ok and delta.total_seconds() < 86400:
|
|
||||||
add("ChatGPT token expiry warning", False, "Expires in < 24h — refresh soon")
|
|
||||||
else:
|
|
||||||
add("ChatGPT token expiry", False, "JWT has no 'exp' claim")
|
|
||||||
except pyjwt.exceptions.DecodeError:
|
|
||||||
# JWE (encrypted JWT) — cannot decode without the server key.
|
|
||||||
# This is normal for ChatGPT's current token format. Token is present and valid.
|
|
||||||
add("ChatGPT token expiry", True, "Encrypted token (JWE) — expiry not decodable client-side")
|
|
||||||
|
|
||||||
# Claude key
|
# Claude key
|
||||||
if claude_key:
|
if claude_key:
|
||||||
@@ -634,6 +612,16 @@ def _print_doctor_table(checks: list[dict]) -> None:
|
|||||||
"(default: images)."
|
"(default: images)."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
@click.option(
|
||||||
|
"--force",
|
||||||
|
is_flag=True,
|
||||||
|
help=(
|
||||||
|
"Re-export every conversation even if already cached and unchanged. "
|
||||||
|
"Use to re-render the whole archive after a formatting/feature change "
|
||||||
|
"without 'cache --clear' (which would drop Joplin note links). Combine "
|
||||||
|
"with --max-conversations to spread the work over several runs."
|
||||||
|
),
|
||||||
|
)
|
||||||
@click.option("--dry-run", is_flag=True, help="Show what would be exported without writing anything.")
|
@click.option("--dry-run", is_flag=True, help="Show what would be exported without writing anything.")
|
||||||
@click.pass_context
|
@click.pass_context
|
||||||
def export(
|
def export(
|
||||||
@@ -646,6 +634,7 @@ def export(
|
|||||||
hidden_content: str | None,
|
hidden_content: str | None,
|
||||||
max_conversations: int | None,
|
max_conversations: int | None,
|
||||||
download_media: str | None,
|
download_media: str | None,
|
||||||
|
force: bool,
|
||||||
dry_run: bool,
|
dry_run: bool,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Export new and updated conversations to Markdown or JSON.
|
"""Export new and updated conversations to Markdown or JSON.
|
||||||
@@ -697,6 +686,18 @@ def export(
|
|||||||
# Session cap: CLI flag wins over MAX_CONVERSATIONS_PER_RUN env default.
|
# Session cap: CLI flag wins over MAX_CONVERSATIONS_PER_RUN env default.
|
||||||
session_cap = max_conversations if max_conversations is not None else cfg.max_conversations
|
session_cap = max_conversations if max_conversations is not None else cfg.max_conversations
|
||||||
|
|
||||||
|
# Force re-render campaign: stamp a start time so progress can be tracked
|
||||||
|
# across runs. A conversation is "done this campaign" once its exported_at
|
||||||
|
# passes the stamp; finished providers then export nothing, and the
|
||||||
|
# "still awaiting" count shrinks to zero. A fresh campaign starts whenever
|
||||||
|
# none is active (the previous one completed or was never started).
|
||||||
|
campaign_at: str | None = None
|
||||||
|
if force and not dry_run:
|
||||||
|
campaign_at = cache.get_force_campaign()
|
||||||
|
if campaign_at is None:
|
||||||
|
campaign_at = datetime.now(timezone.utc).isoformat()
|
||||||
|
cache.set_force_campaign(campaign_at)
|
||||||
|
|
||||||
# Determine which providers to run
|
# Determine which providers to run
|
||||||
providers_to_run = _resolve_providers(provider, cfg)
|
providers_to_run = _resolve_providers(provider, cfg)
|
||||||
if not providers_to_run:
|
if not providers_to_run:
|
||||||
@@ -736,7 +737,9 @@ def export(
|
|||||||
f" [dim]--project filter '{project_filter}': {len(all_convs)} matching conversations.[/dim]"
|
f" [dim]--project filter '{project_filter}': {len(all_convs)} matching conversations.[/dim]"
|
||||||
)
|
)
|
||||||
|
|
||||||
to_export = cache.get_new_or_updated(prov_name, all_convs)
|
to_export = cache.get_new_or_updated(
|
||||||
|
prov_name, all_convs, force=force, campaign_at=campaign_at
|
||||||
|
)
|
||||||
skipped = len(all_convs) - len(to_export)
|
skipped = len(all_convs) - len(to_export)
|
||||||
summary[prov_name]["skipped"] = skipped
|
summary[prov_name]["skipped"] = skipped
|
||||||
|
|
||||||
@@ -758,7 +761,11 @@ def export(
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
if not to_export:
|
if not to_export:
|
||||||
console.print(f" [dim]{skipped} conversations already up to date.[/dim]")
|
done_msg = (
|
||||||
|
"already re-rendered this campaign."
|
||||||
|
if force else "conversations already up to date."
|
||||||
|
)
|
||||||
|
console.print(f" [dim]{skipped} {done_msg}[/dim]")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if deferred:
|
if deferred:
|
||||||
@@ -837,14 +844,38 @@ def export(
|
|||||||
progress.advance(task)
|
progress.advance(task)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
# Force campaign: how many of this provider's conversations still
|
||||||
|
# predate the campaign stamp (i.e. not yet re-rendered this campaign).
|
||||||
|
if force and campaign_at is not None:
|
||||||
|
summary[prov_name]["awaiting"] = sum(
|
||||||
|
1
|
||||||
|
for c in all_convs
|
||||||
|
if cache.exported_at(prov_name, c.get("id") or c.get("uuid", "")) < campaign_at
|
||||||
|
)
|
||||||
|
|
||||||
if not dry_run:
|
if not dry_run:
|
||||||
_print_export_summary(summary)
|
_print_export_summary(summary)
|
||||||
total_deferred = sum(s.get("deferred", 0) for s in summary.values())
|
if force:
|
||||||
if total_deferred:
|
total_awaiting = sum(s.get("awaiting", 0) for s in summary.values())
|
||||||
console.print(
|
exported_now = sum(s.get("exported", 0) for s in summary.values())
|
||||||
f"[yellow]{total_deferred} conversation(s) deferred by the session cap "
|
if total_awaiting:
|
||||||
f"({session_cap} per provider per run). Re-run the same command to continue.[/yellow]"
|
console.print(
|
||||||
)
|
f"[yellow]Force re-render: refreshed {exported_now} this run, "
|
||||||
|
f"{total_awaiting} still to go. Re-run the same command to continue.[/yellow]"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cache.clear_force_campaign()
|
||||||
|
console.print(
|
||||||
|
"[green]Force re-render complete — every conversation has been "
|
||||||
|
"re-rendered with the current code.[/green]"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
total_deferred = sum(s.get("deferred", 0) for s in summary.values())
|
||||||
|
if total_deferred:
|
||||||
|
console.print(
|
||||||
|
f"[yellow]{total_deferred} conversation(s) deferred by the session cap "
|
||||||
|
f"({session_cap} per provider per run). Re-run the same command to continue.[/yellow]"
|
||||||
|
)
|
||||||
# Emit the data-loss summary at INFO level so it lands in the log file
|
# Emit the data-loss summary at INFO level so it lands in the log file
|
||||||
# AND the operator's console (default level is INFO).
|
# AND the operator's console (default level is INFO).
|
||||||
for line in loss_report.format_summary().split("\n"):
|
for line in loss_report.format_summary().split("\n"):
|
||||||
@@ -894,16 +925,18 @@ def _resolve_providers(provider: str, cfg) -> list[tuple[str, object]]:
|
|||||||
try_add("claude", cfg.claude_session_key, ClaudeProvider)
|
try_add("claude", cfg.claude_session_key, ClaudeProvider)
|
||||||
|
|
||||||
if provider in ("claude-code", "all"):
|
if provider in ("claude-code", "all"):
|
||||||
from src.providers.claude_code import ClaudeCodeProvider, DEFAULT_PROJECTS_DIR
|
from src.providers.claude_code import ClaudeCodeProvider, resolve_roots
|
||||||
cc_dir = Path(os.getenv("CLAUDE_CODE_DIR", DEFAULT_PROJECTS_DIR)).expanduser()
|
cc_roots = resolve_roots()
|
||||||
if cc_dir.is_dir():
|
if any(r.is_dir() for r in cc_roots):
|
||||||
result.append((
|
result.append((
|
||||||
"claude-code",
|
"claude-code",
|
||||||
ClaudeCodeProvider(projects_dir=cc_dir, hidden_content=cfg.hidden_content),
|
ClaudeCodeProvider(hidden_content=cfg.hidden_content),
|
||||||
))
|
))
|
||||||
elif provider == "claude-code":
|
elif provider == "claude-code":
|
||||||
logging.getLogger(__name__).warning(
|
logging.getLogger(__name__).warning(
|
||||||
"[claude-code] Skipping — %s not found (set CLAUDE_CODE_DIR).", cc_dir
|
"[claude-code] Skipping — none of these roots found (set "
|
||||||
|
"CLAUDE_CODE_DIR, ':'-separated for multiple): %s",
|
||||||
|
", ".join(str(r) for r in cc_roots),
|
||||||
)
|
)
|
||||||
|
|
||||||
return result
|
return result
|
||||||
@@ -1369,7 +1402,10 @@ def joplin(ctx: click.Context, provider: str, project_filter: str | None, dry_ru
|
|||||||
notebook_id = client.get_or_create_notebook_path(list(nb_path))
|
notebook_id = client.get_or_create_notebook_path(list(nb_path))
|
||||||
|
|
||||||
if existing_note_id:
|
if existing_note_id:
|
||||||
client.update_note(existing_note_id, title, body)
|
# Pass notebook_id so a changed provider→notebook mapping
|
||||||
|
# (e.g. Claude Code AI-Claude → AI-ClaudeCode) relocates
|
||||||
|
# the existing note instead of leaving a stale copy.
|
||||||
|
client.update_note(existing_note_id, title, body, parent_id=notebook_id)
|
||||||
cache_obj.mark_joplin_synced(prov_name, conv_id, existing_note_id)
|
cache_obj.mark_joplin_synced(prov_name, conv_id, existing_note_id)
|
||||||
summary[prov_name]["updated"] += 1
|
summary[prov_name]["updated"] += 1
|
||||||
else:
|
else:
|
||||||
|
|||||||
@@ -50,6 +50,27 @@ VALID_HIDDEN_CONTENT_POLICIES = {
|
|||||||
HIDDEN_CONTENT_OMIT,
|
HIDDEN_CONTENT_OMIT,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# API-drift canary (FUTURE.md §10)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# The export depends on undocumented internal web APIs that can change shape
|
||||||
|
# without notice. The canary fetches a small live sample and asserts only the
|
||||||
|
# normalizer's load-bearing fields (NOT full shape — provider settings objects
|
||||||
|
# churn constantly and would be pure noise). Findings carry a severity:
|
||||||
|
# ERROR — a load-bearing field is missing/mistyped; the parser will break or
|
||||||
|
# silently lose data. The backup is no longer trustworthy.
|
||||||
|
# WARN — something unfamiliar appeared (new content_type, new tool author,
|
||||||
|
# a dormant field went live). Investigate; not necessarily broken.
|
||||||
|
# OK — the assertion held.
|
||||||
|
DRIFT_ERROR = "error"
|
||||||
|
DRIFT_WARN = "warn"
|
||||||
|
DRIFT_OK = "ok"
|
||||||
|
|
||||||
|
|
||||||
|
def drift_finding(severity: str, check: str, detail: str = "") -> dict:
|
||||||
|
"""Build one canary finding."""
|
||||||
|
return {"severity": severity, "check": check, "detail": detail}
|
||||||
|
|
||||||
|
|
||||||
def resolve_hidden_content_policy() -> str:
|
def resolve_hidden_content_policy() -> str:
|
||||||
"""Read EXPORTER_HIDDEN_CONTENT from the environment, defaulting to placeholder."""
|
"""Read EXPORTER_HIDDEN_CONTENT from the environment, defaulting to placeholder."""
|
||||||
@@ -159,6 +180,15 @@ class BaseProvider(ABC):
|
|||||||
isolation, e.g. from tests, doesn't crash).
|
isolation, e.g. from tests, doesn't crash).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
def check_drift(self) -> list[dict]:
|
||||||
|
"""Probe the live API and assert the normalizer's load-bearing fields.
|
||||||
|
|
||||||
|
Returns a list of ``drift_finding`` dicts. The default is a no-op
|
||||||
|
(used by providers with no remote schema risk, e.g. local Claude Code
|
||||||
|
transcripts); web-API providers override this. See FUTURE.md §10.
|
||||||
|
"""
|
||||||
|
return []
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Concrete helpers
|
# Concrete helpers
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|||||||
+148
-5
@@ -45,12 +45,16 @@ from src.blocks import (
|
|||||||
from src.loss_report import LossReport
|
from src.loss_report import LossReport
|
||||||
from src.providers.base import (
|
from src.providers.base import (
|
||||||
BaseProvider,
|
BaseProvider,
|
||||||
|
DRIFT_ERROR,
|
||||||
|
DRIFT_OK,
|
||||||
|
DRIFT_WARN,
|
||||||
HIDDEN_CONTENT_FULL,
|
HIDDEN_CONTENT_FULL,
|
||||||
HIDDEN_CONTENT_OMIT,
|
HIDDEN_CONTENT_OMIT,
|
||||||
HIDDEN_CONTENT_PLACEHOLDER,
|
HIDDEN_CONTENT_PLACEHOLDER,
|
||||||
ProviderError,
|
ProviderError,
|
||||||
REQUEST_TIMEOUT,
|
REQUEST_TIMEOUT,
|
||||||
VALID_HIDDEN_CONTENT_POLICIES,
|
VALID_HIDDEN_CONTENT_POLICIES,
|
||||||
|
drift_finding,
|
||||||
resolve_hidden_content_policy,
|
resolve_hidden_content_policy,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -91,6 +95,20 @@ def parse_asset_file_id(ref: str) -> str | None:
|
|||||||
# myfiles_browser is the legacy name for the same retrieval tool.
|
# myfiles_browser is the legacy name for the same retrieval tool.
|
||||||
_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}
|
_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}
|
||||||
|
|
||||||
|
# ── API-drift canary vocabularies (FUTURE.md §10) ──────────────────────────
|
||||||
|
# Tool authors seen live 2026-06-28. The collapse keys off author.name, so a
|
||||||
|
# RENAMED retrieval tool would silently bypass the collapse and re-bloat the
|
||||||
|
# archive — the canary flags any unfamiliar (role="tool", author.name) pair.
|
||||||
|
_KNOWN_TOOL_AUTHORS = _COLLAPSE_TOOL_AUTHORS | {"web.run", "python"}
|
||||||
|
# content_types the normalizer dispatches on (keep in sync with
|
||||||
|
# _extract_blocks_for_content). A new value already degrades to an `unknown`
|
||||||
|
# block + LossReport at export time; the canary flags it proactively.
|
||||||
|
_HANDLED_CONTENT_TYPES = frozenset({
|
||||||
|
"text", "multimodal_text", "execution_output", "system_error",
|
||||||
|
"tether_browsing_display", "code", "thoughts", "reasoning_recap",
|
||||||
|
"user_editable_context", "model_editable_context", "image_asset_pointer",
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
class ChatGPTProvider(BaseProvider):
|
class ChatGPTProvider(BaseProvider):
|
||||||
"""Provider for ChatGPT conversations via the internal web API.
|
"""Provider for ChatGPT conversations via the internal web API.
|
||||||
@@ -207,19 +225,51 @@ class ChatGPTProvider(BaseProvider):
|
|||||||
len(self._project_ids),
|
len(self._project_ids),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def _get_session_payload(self) -> dict:
|
||||||
|
"""GET /api/auth/session and return the parsed NextAuth JSON.
|
||||||
|
|
||||||
|
Health signal (verified live 2026-06-28): a healthy session returns
|
||||||
|
``accessToken`` with ``error`` absent/null. A *dead* session-token
|
||||||
|
returns HTTP 200 with ``error == "RefreshAccessTokenError"`` and a
|
||||||
|
STALE ``accessToken`` — so ``error``, not token presence, is the
|
||||||
|
reliable validity signal. ``expires`` is a rolling window (it advances
|
||||||
|
on every call even for a dead token) and must NOT be used to judge
|
||||||
|
validity.
|
||||||
|
"""
|
||||||
|
resp = self._session.get(AUTH_SESSION_URL, timeout=REQUEST_TIMEOUT)
|
||||||
|
resp.raise_for_status()
|
||||||
|
return resp.json()
|
||||||
|
|
||||||
|
def session_health(self) -> tuple[bool, str]:
|
||||||
|
"""Return ``(ok, detail)`` describing session-token validity.
|
||||||
|
|
||||||
|
Used by ``doctor`` for a cheap, honest token-health check. Returns
|
||||||
|
``ok=False`` with a refresh hint when NextAuth reports an auth
|
||||||
|
``error`` or omits the access token.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
data = self._get_session_payload()
|
||||||
|
except Exception as e: # network error / non-JSON / HTTP 4xx-5xx
|
||||||
|
return False, f"/api/auth/session unreachable: {str(e)[:60]}"
|
||||||
|
error = data.get("error")
|
||||||
|
if error:
|
||||||
|
return False, (
|
||||||
|
f"Session expired (error={error}) — refresh "
|
||||||
|
"__Secure-next-auth.session-token via DevTools / 'auth'"
|
||||||
|
)
|
||||||
|
if not data.get("accessToken"):
|
||||||
|
return False, "No accessToken — refresh token via DevTools / 'auth'"
|
||||||
|
return True, "Session active"
|
||||||
|
|
||||||
def _fetch_access_token(self) -> str:
|
def _fetch_access_token(self) -> str:
|
||||||
"""Exchange the session cookie for a Bearer access token.
|
"""Exchange the session cookie for a Bearer access token.
|
||||||
|
|
||||||
Calls GET /api/auth/session — the cookie jar already contains the
|
Calls GET /api/auth/session — the cookie jar already contains the
|
||||||
session token, so no manual Cookie header is needed.
|
session token, so no manual Cookie header is needed.
|
||||||
|
|
||||||
Returns {"accessToken": "...", "user": {...}}.
|
|
||||||
"""
|
"""
|
||||||
logger.debug("[chatgpt] Fetching access token from %s", AUTH_SESSION_URL)
|
logger.debug("[chatgpt] Fetching access token from %s", AUTH_SESSION_URL)
|
||||||
try:
|
try:
|
||||||
resp = self._session.get(AUTH_SESSION_URL, timeout=REQUEST_TIMEOUT)
|
data = self._get_session_payload()
|
||||||
resp.raise_for_status()
|
|
||||||
data = resp.json()
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise ProviderError(
|
raise ProviderError(
|
||||||
self.provider_name,
|
self.provider_name,
|
||||||
@@ -230,6 +280,21 @@ class ChatGPTProvider(BaseProvider):
|
|||||||
),
|
),
|
||||||
) from e
|
) from e
|
||||||
|
|
||||||
|
# A dead session-token returns HTTP 200 with error=RefreshAccessTokenError
|
||||||
|
# and a stale accessToken. Fail fast here rather than letting the stale
|
||||||
|
# token produce a confusing downstream 401.
|
||||||
|
error = data.get("error")
|
||||||
|
if error:
|
||||||
|
raise ProviderError(
|
||||||
|
self.provider_name,
|
||||||
|
"fetch_access_token",
|
||||||
|
RuntimeError(
|
||||||
|
f"Session token expired (/api/auth/session error={error!r}). "
|
||||||
|
"Refresh __Secure-next-auth.session-token via DevTools, then run "
|
||||||
|
"'ai-chat-exporter auth' or update CHATGPT_SESSION_TOKEN in .env."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
access_token = data.get("accessToken")
|
access_token = data.get("accessToken")
|
||||||
if not access_token:
|
if not access_token:
|
||||||
raise ProviderError(
|
raise ProviderError(
|
||||||
@@ -719,6 +784,84 @@ class ChatGPTProvider(BaseProvider):
|
|||||||
"messages": messages,
|
"messages": messages,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
def check_drift(self) -> list[dict]:
|
||||||
|
"""Probe one listing page + one conversation; assert load-bearing fields.
|
||||||
|
|
||||||
|
See FUTURE.md §10. Asserts only the fields the normalizer depends on —
|
||||||
|
not the full response shape (which churns harmlessly). The top silent
|
||||||
|
risk is a renamed retrieval tool author bypassing the collapse.
|
||||||
|
"""
|
||||||
|
findings: list[dict] = []
|
||||||
|
page = self.list_conversations(offset=0, limit=5)
|
||||||
|
if not page:
|
||||||
|
return [drift_finding(DRIFT_WARN, "listing",
|
||||||
|
"empty listing — cannot verify shape")]
|
||||||
|
|
||||||
|
item = page[0]
|
||||||
|
for f in ("id", "title"):
|
||||||
|
if f not in item:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "listing", f"summary item missing '{f}'"))
|
||||||
|
if not (item.get("update_time") or item.get("create_time")):
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "listing", "no update_time/create_time on summary"))
|
||||||
|
|
||||||
|
conv_id = item.get("id")
|
||||||
|
if not conv_id:
|
||||||
|
return findings or [drift_finding(DRIFT_ERROR, "listing", "no id to fetch")]
|
||||||
|
raw = self.get_conversation(conv_id)
|
||||||
|
if not (raw.get("conversation_id") or raw.get("id")):
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "detail", "no conversation_id/id on detail"))
|
||||||
|
mapping = raw.get("mapping")
|
||||||
|
if not isinstance(mapping, dict) or not mapping:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "detail", "mapping missing/empty — tree walk will yield 0 messages"))
|
||||||
|
return findings
|
||||||
|
|
||||||
|
saw_text_with_parts = False
|
||||||
|
seen_new_ct: set[str] = set()
|
||||||
|
seen_new_tool: set[str] = set()
|
||||||
|
for node in mapping.values():
|
||||||
|
if "children" not in node:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "mapping", "node missing 'children' — tree walk breaks"))
|
||||||
|
break
|
||||||
|
msg = node.get("message")
|
||||||
|
if not msg:
|
||||||
|
continue
|
||||||
|
author = msg.get("author") or {}
|
||||||
|
role, name = author.get("role"), author.get("name")
|
||||||
|
content = msg.get("content") or {}
|
||||||
|
ct = content.get("content_type")
|
||||||
|
if ct and ct not in _HANDLED_CONTENT_TYPES:
|
||||||
|
seen_new_ct.add(ct)
|
||||||
|
if role == "tool" and name and name not in _KNOWN_TOOL_AUTHORS:
|
||||||
|
seen_new_tool.add(name)
|
||||||
|
if ct == "text":
|
||||||
|
parts = content.get("parts") or []
|
||||||
|
if any(isinstance(p, str) and p.strip() for p in parts):
|
||||||
|
saw_text_with_parts = True
|
||||||
|
|
||||||
|
for name in sorted(seen_new_tool):
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "collapse",
|
||||||
|
f"unfamiliar tool author {name!r} — if it's a retrieval dump it "
|
||||||
|
"is NOT being collapsed (silent archive bloat); update "
|
||||||
|
"_COLLAPSE_TOOL_AUTHORS / _KNOWN_TOOL_AUTHORS"))
|
||||||
|
for ct in sorted(seen_new_ct):
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "content_type",
|
||||||
|
f"new content_type {ct!r} — exported as an unknown block; add a handler"))
|
||||||
|
if not saw_text_with_parts:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "parts",
|
||||||
|
"no text message yielded non-empty parts — content.parts may have drifted"))
|
||||||
|
|
||||||
|
if not findings:
|
||||||
|
findings.append(drift_finding(DRIFT_OK, "schema", "all load-bearing fields present"))
|
||||||
|
return findings
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Internal helpers
|
# Internal helpers
|
||||||
|
|||||||
+72
-1
@@ -16,7 +16,14 @@ from src.blocks import (
|
|||||||
make_unknown_block,
|
make_unknown_block,
|
||||||
)
|
)
|
||||||
from src.loss_report import LossReport
|
from src.loss_report import LossReport
|
||||||
from src.providers.base import BaseProvider, ProviderError
|
from src.providers.base import (
|
||||||
|
BaseProvider,
|
||||||
|
DRIFT_ERROR,
|
||||||
|
DRIFT_OK,
|
||||||
|
DRIFT_WARN,
|
||||||
|
ProviderError,
|
||||||
|
drift_finding,
|
||||||
|
)
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -232,6 +239,70 @@ class ClaudeProvider(BaseProvider):
|
|||||||
"messages": messages,
|
"messages": messages,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
def check_drift(self) -> list[dict]:
|
||||||
|
"""Probe one listing page + one conversation; assert load-bearing fields.
|
||||||
|
|
||||||
|
See FUTURE.md §10. Real Claude messages are flat ``text``/``sender``;
|
||||||
|
the rich ``content`` block path is dormant, so its activation (content
|
||||||
|
becoming a list) is itself a drift event worth flagging. ``attachments``
|
||||||
|
/ ``files`` are ignored by the normalizer — non-empty values are a
|
||||||
|
silent-loss risk.
|
||||||
|
"""
|
||||||
|
findings: list[dict] = []
|
||||||
|
page = self.list_conversations(offset=0, limit=5)
|
||||||
|
if not page:
|
||||||
|
return [drift_finding(DRIFT_WARN, "listing",
|
||||||
|
"empty listing — cannot verify shape")]
|
||||||
|
|
||||||
|
item = page[0]
|
||||||
|
if not (item.get("uuid") or item.get("id")):
|
||||||
|
findings.append(drift_finding(DRIFT_ERROR, "listing", "summary missing uuid/id"))
|
||||||
|
if not (item.get("name") or item.get("title")):
|
||||||
|
findings.append(drift_finding(DRIFT_WARN, "listing", "no name/title on summary"))
|
||||||
|
if not (item.get("updated_at") or item.get("update_time")):
|
||||||
|
findings.append(drift_finding(DRIFT_WARN, "listing", "no updated_at on summary"))
|
||||||
|
|
||||||
|
conv_id = item.get("uuid") or item.get("id")
|
||||||
|
if not conv_id:
|
||||||
|
return findings or [drift_finding(DRIFT_ERROR, "listing", "no id to fetch")]
|
||||||
|
raw = self.get_conversation(conv_id)
|
||||||
|
if not (raw.get("uuid") or raw.get("id")):
|
||||||
|
findings.append(drift_finding(DRIFT_ERROR, "detail", "no uuid/id on detail"))
|
||||||
|
msgs = raw.get("chat_messages") or raw.get("messages")
|
||||||
|
if not isinstance(msgs, list) or not msgs:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_ERROR, "detail", "chat_messages missing/empty — 0 messages"))
|
||||||
|
return findings
|
||||||
|
|
||||||
|
saw_sender = saw_text = False
|
||||||
|
flagged_list = flagged_attach = False
|
||||||
|
for m in msgs:
|
||||||
|
if m.get("sender") or m.get("role"):
|
||||||
|
saw_sender = True
|
||||||
|
content = m.get("content")
|
||||||
|
if isinstance(content, list) and not flagged_list:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "content",
|
||||||
|
"message.content is now a LIST — Claude switched to typed blocks; "
|
||||||
|
"the dormant _dispatch_claude_block path is now live, verify rendering"))
|
||||||
|
flagged_list = True
|
||||||
|
if m.get("text") or content:
|
||||||
|
saw_text = True
|
||||||
|
if (m.get("attachments") or m.get("files")) and not flagged_attach:
|
||||||
|
findings.append(drift_finding(
|
||||||
|
DRIFT_WARN, "attachments",
|
||||||
|
"message has non-empty attachments/files — the normalizer ignores "
|
||||||
|
"these (silent loss); add handling"))
|
||||||
|
flagged_attach = True
|
||||||
|
|
||||||
|
if not saw_sender:
|
||||||
|
findings.append(drift_finding(DRIFT_ERROR, "messages", "no sender/role on any message"))
|
||||||
|
if not saw_text:
|
||||||
|
findings.append(drift_finding(DRIFT_WARN, "messages", "no text/content found"))
|
||||||
|
if not findings:
|
||||||
|
findings.append(drift_finding(DRIFT_OK, "schema", "all load-bearing fields present"))
|
||||||
|
return findings
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Internal helpers
|
# Internal helpers
|
||||||
|
|||||||
+331
-57
@@ -16,9 +16,21 @@ Record types in a session file: ``user`` / ``assistant`` carry the dialogue
|
|||||||
(Anthropic-style ``message.content`` block arrays); ``ai-title`` carries the
|
(Anthropic-style ``message.content`` block arrays); ``ai-title`` carries the
|
||||||
evolving session title (last one wins); ``last-prompt``,
|
evolving session title (last one wins); ``last-prompt``,
|
||||||
``file-history-snapshot``, ``attachment``, ``permission-mode``, ``system``
|
``file-history-snapshot``, ``attachment``, ``permission-mode``, ``system``
|
||||||
are harness records and are skipped. Records flagged ``isSidechain`` are
|
are harness records and are skipped. ``isMeta`` records are harness-generated
|
||||||
subagent transcripts; ``isMeta`` are harness-generated user records — both
|
user records and are skipped.
|
||||||
skipped.
|
|
||||||
|
Subagents (Task tool): Claude Code stores each subagent's transcript as a
|
||||||
|
separate ``<session>/subagents/agent-*.jsonl`` file with an ``agent-*.meta.json``
|
||||||
|
sidecar (``agentType``, ``description``, ``toolUseId``). ``_load_subagents``
|
||||||
|
loads them keyed by ``toolUseId``; at the ``Task``/``Agent`` tool call that
|
||||||
|
spawned it, the subagent is folded inline as a collapsible ``<details>`` block
|
||||||
|
(the subagent's own records are flagged ``isSidechain`` and only processed on
|
||||||
|
this recursive pass — the top-level pass still skips sidechain records).
|
||||||
|
|
||||||
|
Scan scope: multiple ``projects/`` roots are supported — ``CLAUDE_CODE_DIR`` may
|
||||||
|
be an ``os.pathsep``-separated list and ``CLAUDE_CONFIG_DIR``'s ``projects/`` is
|
||||||
|
included when set. Sessions from all roots are merged by launch-folder; a session
|
||||||
|
UUID present in two roots keeps the newer-mtime copy. See ``resolve_roots``.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
@@ -34,6 +46,7 @@ from src.blocks import (
|
|||||||
UNKNOWN_REASON_UNKNOWN_TYPE,
|
UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||||
make_collapsed_block,
|
make_collapsed_block,
|
||||||
make_image_placeholder,
|
make_image_placeholder,
|
||||||
|
make_subagent_block,
|
||||||
make_text_block,
|
make_text_block,
|
||||||
make_thinking_block,
|
make_thinking_block,
|
||||||
make_tool_result_block,
|
make_tool_result_block,
|
||||||
@@ -53,6 +66,47 @@ logger = logging.getLogger(__name__)
|
|||||||
|
|
||||||
DEFAULT_PROJECTS_DIR = "~/.claude/projects"
|
DEFAULT_PROJECTS_DIR = "~/.claude/projects"
|
||||||
|
|
||||||
|
# Tool names whose call spawns a subagent (Task tool) — its separate transcript
|
||||||
|
# is folded inline. Both spellings have appeared across Claude Code versions.
|
||||||
|
_SUBAGENT_TOOL_NAMES = {"Task", "Agent"}
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_roots(projects_dir=None) -> list[Path]:
|
||||||
|
"""Resolve the ordered list of Claude Code ``projects/`` roots to scan.
|
||||||
|
|
||||||
|
Precedence:
|
||||||
|
1. Explicit ``projects_dir`` (a single path or a list) — used by tests.
|
||||||
|
2. ``CLAUDE_CODE_DIR`` env, split on ``os.pathsep`` (``:``) so multiple
|
||||||
|
roots can be given; a single path (no separator) stays backward
|
||||||
|
compatible. Falls back to the default root when unset.
|
||||||
|
3. Additionally, if ``CLAUDE_CONFIG_DIR`` is set, its ``projects/`` subdir
|
||||||
|
is appended — this is the common "non-default config dir" case.
|
||||||
|
|
||||||
|
Roots are expanded, de-duplicated (order preserved), and returned as-is
|
||||||
|
(existence is checked by the caller / ``_scan``). Sessions from all roots are
|
||||||
|
merged by launch-folder; there is no per-source label.
|
||||||
|
"""
|
||||||
|
raw: list[str]
|
||||||
|
if projects_dir is not None:
|
||||||
|
raw = [str(p) for p in projects_dir] if isinstance(projects_dir, (list, tuple)) \
|
||||||
|
else [str(projects_dir)]
|
||||||
|
else:
|
||||||
|
env = os.getenv("CLAUDE_CODE_DIR")
|
||||||
|
raw = env.split(os.pathsep) if env else [DEFAULT_PROJECTS_DIR]
|
||||||
|
config_dir = os.getenv("CLAUDE_CONFIG_DIR")
|
||||||
|
if config_dir:
|
||||||
|
raw.append(str(Path(config_dir) / "projects"))
|
||||||
|
|
||||||
|
roots: list[Path] = []
|
||||||
|
for p in raw:
|
||||||
|
p = p.strip()
|
||||||
|
if not p:
|
||||||
|
continue
|
||||||
|
path = Path(p).expanduser()
|
||||||
|
if path not in roots:
|
||||||
|
roots.append(path)
|
||||||
|
return roots
|
||||||
|
|
||||||
# Harness-injected tags inside user message text. Stripped so exports contain
|
# Harness-injected tags inside user message text. Stripped so exports contain
|
||||||
# the dialogue, not the CLI plumbing. A record that is nothing but tags
|
# the dialogue, not the CLI plumbing. A record that is nothing but tags
|
||||||
# (e.g. a /model invocation) ends up empty and is skipped.
|
# (e.g. a /model invocation) ends up empty and is skipped.
|
||||||
@@ -88,9 +142,7 @@ class ClaudeCodeProvider(BaseProvider):
|
|||||||
hidden_content: str | None = None,
|
hidden_content: str | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
super().__init__()
|
super().__init__()
|
||||||
self._projects_dir = Path(
|
self._projects_dirs = resolve_roots(projects_dir)
|
||||||
projects_dir or os.getenv("CLAUDE_CODE_DIR", DEFAULT_PROJECTS_DIR)
|
|
||||||
).expanduser()
|
|
||||||
self._hidden_content = (
|
self._hidden_content = (
|
||||||
hidden_content
|
hidden_content
|
||||||
if hidden_content in VALID_HIDDEN_CONTENT_POLICIES
|
if hidden_content in VALID_HIDDEN_CONTENT_POLICIES
|
||||||
@@ -116,7 +168,9 @@ class ClaudeCodeProvider(BaseProvider):
|
|||||||
if datetime.fromisoformat(c["updated_at"]) >= since_aware
|
if datetime.fromisoformat(c["updated_at"]) >= since_aware
|
||||||
]
|
]
|
||||||
logger.info(
|
logger.info(
|
||||||
"[claude-code] Found %d session(s) under %s", len(convs), self._projects_dir
|
"[claude-code] Found %d session(s) under %s",
|
||||||
|
len(convs),
|
||||||
|
", ".join(str(d) for d in self._projects_dirs),
|
||||||
)
|
)
|
||||||
return convs
|
return convs
|
||||||
|
|
||||||
@@ -133,26 +187,16 @@ class ClaudeCodeProvider(BaseProvider):
|
|||||||
FileNotFoundError(f"No session file for id {conv_id}"),
|
FileNotFoundError(f"No session file for id {conv_id}"),
|
||||||
)
|
)
|
||||||
|
|
||||||
records: list[dict] = []
|
records = _parse_jsonl(path)
|
||||||
bad_lines = 0
|
|
||||||
for line in path.read_text(encoding="utf-8").splitlines():
|
|
||||||
line = line.strip()
|
|
||||||
if not line:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
records.append(json.loads(line))
|
|
||||||
except json.JSONDecodeError:
|
|
||||||
bad_lines += 1
|
|
||||||
if bad_lines:
|
|
||||||
logger.warning(
|
|
||||||
"[claude-code] %s: skipped %d unparseable line(s)", path.name, bad_lines
|
|
||||||
)
|
|
||||||
|
|
||||||
mtime = datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc)
|
mtime = datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc)
|
||||||
return {
|
return {
|
||||||
"id": conv_id,
|
"id": conv_id,
|
||||||
"_path": str(path),
|
"_path": str(path),
|
||||||
"_records": records,
|
"_records": records,
|
||||||
|
# Subagent (Task-tool) transcripts, keyed by the parent tool_use id
|
||||||
|
# that spawned them, folded inline during normalization.
|
||||||
|
"_subagents": _load_subagents(path),
|
||||||
# Listing and normalized updated_at must match, or the cache
|
# Listing and normalized updated_at must match, or the cache
|
||||||
# staleness comparison would re-export every session every run.
|
# staleness comparison would re-export every session every run.
|
||||||
"_mtime_iso": mtime.isoformat(),
|
"_mtime_iso": mtime.isoformat(),
|
||||||
@@ -164,7 +208,16 @@ class ClaudeCodeProvider(BaseProvider):
|
|||||||
conv_id = raw.get("id") or ""
|
conv_id = raw.get("id") or ""
|
||||||
records: list[dict] = raw.get("_records") or []
|
records: list[dict] = raw.get("_records") or []
|
||||||
|
|
||||||
|
subagents: dict = raw.get("_subagents") or {}
|
||||||
title = _extract_title(records)
|
title = _extract_title(records)
|
||||||
|
# Append the repos this session touched, e.g. "… [repo-a, repo-b]", so
|
||||||
|
# sessions launched from a workspace root (which all land in one
|
||||||
|
# folder-named notebook) stay scannable and searchable.
|
||||||
|
launch_cwd = _extract_launch_cwd(records)
|
||||||
|
if launch_cwd:
|
||||||
|
repos = _repos_touched(records, launch_cwd)
|
||||||
|
if repos:
|
||||||
|
title = f"{title} [{', '.join(repos)}]"
|
||||||
project = _extract_project(records, raw.get("_path"))
|
project = _extract_project(records, raw.get("_path"))
|
||||||
created_at = next(
|
created_at = next(
|
||||||
(r.get("timestamp") for r in records if r.get("timestamp")), ""
|
(r.get("timestamp") for r in records if r.get("timestamp")), ""
|
||||||
@@ -173,7 +226,7 @@ class ClaudeCodeProvider(BaseProvider):
|
|||||||
(r.get("timestamp") for r in reversed(records) if r.get("timestamp")), ""
|
(r.get("timestamp") for r in reversed(records) if r.get("timestamp")), ""
|
||||||
)
|
)
|
||||||
|
|
||||||
messages = _extract_messages(records, conv_id, report, policy)
|
messages = _extract_messages(records, conv_id, report, policy, subagents)
|
||||||
for _ in messages:
|
for _ in messages:
|
||||||
report.record_message()
|
report.record_message()
|
||||||
report.record_conversation()
|
report.record_conversation()
|
||||||
@@ -194,41 +247,59 @@ class ClaudeCodeProvider(BaseProvider):
|
|||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
def _scan(self) -> list[dict]:
|
def _scan(self) -> list[dict]:
|
||||||
if not self._projects_dir.is_dir():
|
existing = [d for d in self._projects_dirs if d.is_dir()]
|
||||||
|
if not existing:
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"[claude-code] Projects directory %s does not exist", self._projects_dir
|
"[claude-code] No projects directory exists: %s",
|
||||||
|
", ".join(str(d) for d in self._projects_dirs),
|
||||||
)
|
)
|
||||||
return []
|
return []
|
||||||
|
|
||||||
convs: list[dict] = []
|
# conv_id → (mtime, conv dict). Roots are merged by launch-folder; if the
|
||||||
for proj_dir in sorted(p for p in self._projects_dir.iterdir() if p.is_dir()):
|
# same session UUID appears in two roots (e.g. a live dir and a backup),
|
||||||
for session_file in sorted(proj_dir.glob("*.jsonl")):
|
# the newer-mtime copy wins so we never emit two entries for one session.
|
||||||
try:
|
by_id: dict[str, tuple[float, dict]] = {}
|
||||||
stat = session_file.stat()
|
for root in existing:
|
||||||
except OSError:
|
for proj_dir in sorted(p for p in root.iterdir() if p.is_dir()):
|
||||||
continue
|
for session_file in sorted(proj_dir.glob("*.jsonl")):
|
||||||
if stat.st_size == 0:
|
try:
|
||||||
continue
|
stat = session_file.stat()
|
||||||
conv_id = session_file.stem
|
except OSError:
|
||||||
self._path_map[conv_id] = session_file
|
continue
|
||||||
title, project, created = _read_session_meta(session_file)
|
if stat.st_size == 0:
|
||||||
convs.append(
|
continue
|
||||||
{
|
conv_id = session_file.stem
|
||||||
"id": conv_id,
|
prev = by_id.get(conv_id)
|
||||||
"title": title,
|
if prev is not None and prev[0] >= stat.st_mtime:
|
||||||
"project": project,
|
logger.debug(
|
||||||
# The --project filter and dry-run table read the
|
"[claude-code] Duplicate session %s in %s; keeping newer copy",
|
||||||
# listing dict, not the normalized conversation.
|
conv_id[:8], root,
|
||||||
"_project_name": project,
|
)
|
||||||
"created_at": created,
|
continue
|
||||||
"updated_at": datetime.fromtimestamp(
|
self._path_map[conv_id] = session_file
|
||||||
stat.st_mtime, tz=timezone.utc
|
title, project, created = _read_session_meta(session_file)
|
||||||
).isoformat(),
|
by_id[conv_id] = (
|
||||||
"_path": str(session_file),
|
stat.st_mtime,
|
||||||
"_project_dir": proj_dir.name,
|
{
|
||||||
}
|
"id": conv_id,
|
||||||
)
|
"title": title,
|
||||||
return convs
|
"project": project,
|
||||||
|
# The --project filter and dry-run table read the
|
||||||
|
# listing dict, not the normalized conversation.
|
||||||
|
"_project_name": project,
|
||||||
|
"created_at": created,
|
||||||
|
"updated_at": datetime.fromtimestamp(
|
||||||
|
stat.st_mtime, tz=timezone.utc
|
||||||
|
).isoformat(),
|
||||||
|
"_path": str(session_file),
|
||||||
|
"_project_dir": proj_dir.name,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
# Deterministic order: by bucket dir, then session id.
|
||||||
|
return sorted(
|
||||||
|
(conv for _, conv in by_id.values()),
|
||||||
|
key=lambda c: (c["_project_dir"], c["id"]),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -315,9 +386,188 @@ def _extract_project(records: list[dict], path: str | None) -> str | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_jsonl(path: Path) -> list[dict]:
|
||||||
|
"""Read a JSONL file into records, tolerating (and logging) bad lines."""
|
||||||
|
records: list[dict] = []
|
||||||
|
bad_lines = 0
|
||||||
|
try:
|
||||||
|
text = path.read_text(encoding="utf-8")
|
||||||
|
except OSError as e:
|
||||||
|
logger.warning("[claude-code] Could not read %s: %s", path, e)
|
||||||
|
return records
|
||||||
|
for line in text.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
records.append(json.loads(line))
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
bad_lines += 1
|
||||||
|
if bad_lines:
|
||||||
|
logger.warning(
|
||||||
|
"[claude-code] %s: skipped %d unparseable line(s)", path.name, bad_lines
|
||||||
|
)
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def _load_subagents(session_path: Path) -> dict:
|
||||||
|
"""Map spawning tool_use id → ``{"meta": {...}, "records": [...]}``.
|
||||||
|
|
||||||
|
Claude Code writes subagent transcripts to
|
||||||
|
``<session>/subagents/agent-*.jsonl`` with an ``agent-*.meta.json`` sidecar
|
||||||
|
carrying ``toolUseId`` (which parent ``Task``/``Agent`` call spawned it).
|
||||||
|
Files without a readable meta (no id to position them) are skipped.
|
||||||
|
"""
|
||||||
|
submap: dict[str, dict] = {}
|
||||||
|
subdir = session_path.parent / session_path.stem / "subagents"
|
||||||
|
if not subdir.is_dir():
|
||||||
|
return submap
|
||||||
|
for jf in sorted(subdir.glob("*.jsonl")):
|
||||||
|
meta_path = jf.with_suffix(".meta.json")
|
||||||
|
try:
|
||||||
|
meta = json.loads(meta_path.read_text(encoding="utf-8"))
|
||||||
|
except (OSError, json.JSONDecodeError):
|
||||||
|
logger.warning(
|
||||||
|
"[claude-code] Subagent %s has no readable meta; skipping", jf.name
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
tool_id = meta.get("toolUseId")
|
||||||
|
if not tool_id:
|
||||||
|
continue
|
||||||
|
submap[tool_id] = {"meta": meta, "records": _parse_jsonl(jf)}
|
||||||
|
return submap
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_launch_cwd(records: list[dict]) -> str | None:
|
||||||
|
"""The session's working directory (constant per session; first cwd wins)."""
|
||||||
|
for rec in records:
|
||||||
|
cwd = rec.get("cwd")
|
||||||
|
if cwd:
|
||||||
|
return cwd
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# tool_use input keys that carry a file path.
|
||||||
|
_TOOL_PATH_KEYS = ("file_path", "path", "notebook_path")
|
||||||
|
|
||||||
|
# Optional ignore-list: git repos to never tag (comma-separated names). Rarely
|
||||||
|
# needed with git-root detection (config/reference dirs are already excluded
|
||||||
|
# because they aren't git repos), but kept as an escape hatch.
|
||||||
|
def _ignored_repos() -> set[str]:
|
||||||
|
env = os.getenv("CLAUDE_CODE_REPO_TAG_IGNORE", "")
|
||||||
|
return {s.strip() for s in env.split(",") if s.strip()}
|
||||||
|
|
||||||
|
|
||||||
|
# dir Path → git-repo name it belongs to (or None). Process-wide; the working
|
||||||
|
# tree doesn't change under us mid-run, so caching walked dirs is safe.
|
||||||
|
_GIT_ROOT_CACHE: dict[Path, str | None] = {}
|
||||||
|
_CACHE_MISS = object()
|
||||||
|
|
||||||
|
|
||||||
|
def _git_root_name(path: Path, max_steps: int = 25) -> str | None:
|
||||||
|
"""Name of the git repo ``path`` lives in — nearest ancestor with ``.git``.
|
||||||
|
|
||||||
|
Walks up from ``path`` until a ``.git`` entry is found (returns that dir's
|
||||||
|
basename) or the filesystem root is reached (returns ``None``). Disk-based:
|
||||||
|
a path in no git repo, or a repo no longer on disk, yields ``None``.
|
||||||
|
"""
|
||||||
|
cur = path
|
||||||
|
for _ in range(max_steps):
|
||||||
|
cached = _GIT_ROOT_CACHE.get(cur, _CACHE_MISS)
|
||||||
|
if cached is not _CACHE_MISS:
|
||||||
|
return cached
|
||||||
|
try:
|
||||||
|
if (cur / ".git").exists():
|
||||||
|
_GIT_ROOT_CACHE[cur] = cur.name
|
||||||
|
return cur.name
|
||||||
|
except OSError:
|
||||||
|
break
|
||||||
|
if cur.parent == cur: # filesystem root
|
||||||
|
break
|
||||||
|
cur = cur.parent
|
||||||
|
_GIT_ROOT_CACHE[path] = None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# Absolute path-like tokens inside Bash command strings (file_path/path keys are
|
||||||
|
# matched directly). git-root resolution short-circuits on non-repo paths.
|
||||||
|
_ABS_PATH_RE = re.compile(r"/(?:[\w.\-]+/)*[\w.\-]+")
|
||||||
|
|
||||||
|
|
||||||
|
def _repos_touched(
|
||||||
|
records: list[dict], launch_cwd: str, cap: int = 3, ignore: set[str] | None = None
|
||||||
|
) -> list[str]:
|
||||||
|
"""Repos a session touched, for the title's ``[repo-a, repo-b]`` tag.
|
||||||
|
|
||||||
|
A file's repo is the **git repository it lives in** (nearest ancestor with a
|
||||||
|
``.git``), resolved for every absolute path seen in a tool_use input
|
||||||
|
(``file_path``/``path``/``notebook_path`` and absolute home paths inside Bash
|
||||||
|
``command`` strings). This works anywhere in the filesystem — not just under
|
||||||
|
the launch directory — so cross-workspace work is captured, and non-repo
|
||||||
|
noise (config dirs, one-off files, reference dirs) is excluded because it
|
||||||
|
isn't a git repo. Ordered by touch frequency, capped with a trailing ``…``.
|
||||||
|
"""
|
||||||
|
ignore = _ignored_repos() if ignore is None else ignore
|
||||||
|
counts: Counter = Counter()
|
||||||
|
seen: set[str] = set()
|
||||||
|
|
||||||
|
def note(p) -> None:
|
||||||
|
if not isinstance(p, str) or not p:
|
||||||
|
return
|
||||||
|
if not p.startswith("/"): # resolve relative paths against the launch cwd
|
||||||
|
if not launch_cwd:
|
||||||
|
return
|
||||||
|
p = str(Path(launch_cwd) / p)
|
||||||
|
if p in seen:
|
||||||
|
return
|
||||||
|
seen.add(p)
|
||||||
|
name = _git_root_name(Path(p))
|
||||||
|
if name and not name.startswith(".") and name not in ignore:
|
||||||
|
counts[name] += 1
|
||||||
|
|
||||||
|
for rec in records:
|
||||||
|
msg = rec.get("message") or {}
|
||||||
|
content = msg.get("content")
|
||||||
|
if not isinstance(content, list):
|
||||||
|
continue
|
||||||
|
for item in content:
|
||||||
|
if not isinstance(item, dict) or item.get("type") != "tool_use":
|
||||||
|
continue
|
||||||
|
inp = item.get("input")
|
||||||
|
if not isinstance(inp, dict):
|
||||||
|
continue
|
||||||
|
for key in _TOOL_PATH_KEYS:
|
||||||
|
note(inp.get(key))
|
||||||
|
cmd = inp.get("command")
|
||||||
|
if isinstance(cmd, str):
|
||||||
|
for m in _ABS_PATH_RE.finditer(cmd):
|
||||||
|
note(m.group(0))
|
||||||
|
|
||||||
|
ordered = [name for name, _ in counts.most_common()]
|
||||||
|
if len(ordered) > cap:
|
||||||
|
return ordered[:cap] + ["…"]
|
||||||
|
return ordered
|
||||||
|
|
||||||
|
|
||||||
def _extract_messages(
|
def _extract_messages(
|
||||||
records: list[dict], conv_id: str, report: LossReport, policy: str
|
records: list[dict],
|
||||||
|
conv_id: str,
|
||||||
|
report: LossReport,
|
||||||
|
policy: str,
|
||||||
|
subagents: dict | None = None,
|
||||||
|
include_sidechain: bool = False,
|
||||||
) -> list[dict]:
|
) -> list[dict]:
|
||||||
|
"""Normalize Claude Code records into messages.
|
||||||
|
|
||||||
|
``subagents`` maps a spawning ``Task``/``Agent`` tool_use id to its separate
|
||||||
|
transcript ``{"meta": ..., "records": ...}``; when a matching tool_use is
|
||||||
|
seen its subagent is folded inline as a subagent block (extracted
|
||||||
|
recursively under the same policy). ``include_sidechain`` is set True for
|
||||||
|
those recursive subagent passes (subagent records are flagged
|
||||||
|
``isSidechain``); the top-level pass keeps skipping sidechain records so a
|
||||||
|
subagent is never also emitted as a stray top-level turn.
|
||||||
|
"""
|
||||||
|
subagents = subagents or {}
|
||||||
messages: list[dict] = []
|
messages: list[dict] = []
|
||||||
# Pending collapsed tool activity: name → call count, plus total bytes.
|
# Pending collapsed tool activity: name → call count, plus total bytes.
|
||||||
pending_tools: Counter = Counter()
|
pending_tools: Counter = Counter()
|
||||||
@@ -351,7 +601,9 @@ def _extract_messages(
|
|||||||
for rec in records:
|
for rec in records:
|
||||||
if rec.get("type") not in ("user", "assistant"):
|
if rec.get("type") not in ("user", "assistant"):
|
||||||
continue
|
continue
|
||||||
if rec.get("isSidechain") or rec.get("isMeta"):
|
if rec.get("isMeta"):
|
||||||
|
continue
|
||||||
|
if rec.get("isSidechain") and not include_sidechain:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
msg = rec.get("message") or {}
|
msg = rec.get("message") or {}
|
||||||
@@ -392,14 +644,36 @@ def _extract_messages(
|
|||||||
"thinking", len(json.dumps(item, default=str))
|
"thinking", len(json.dumps(item, default=str))
|
||||||
)
|
)
|
||||||
elif item_type == "tool_use":
|
elif item_type == "tool_use":
|
||||||
if policy == HIDDEN_CONTENT_FULL:
|
name = item.get("name") or "tool"
|
||||||
|
tool_id = item.get("id")
|
||||||
|
if name in _SUBAGENT_TOOL_NAMES and tool_id in subagents:
|
||||||
|
# A Task/Agent spawn: fold its separate transcript inline
|
||||||
|
# instead of collapsing it. Its own tool traffic is
|
||||||
|
# collapsed by the recursive pass under the same policy.
|
||||||
|
sub = subagents[tool_id]
|
||||||
|
meta = sub.get("meta") or {}
|
||||||
|
sub_msgs = _extract_messages(
|
||||||
|
sub.get("records") or [],
|
||||||
|
conv_id,
|
||||||
|
report,
|
||||||
|
policy,
|
||||||
|
subagents,
|
||||||
|
include_sidechain=True,
|
||||||
|
)
|
||||||
|
blocks.append(
|
||||||
|
make_subagent_block(
|
||||||
|
agent_type=meta.get("agentType") or name,
|
||||||
|
description=meta.get("description") or "",
|
||||||
|
messages=sub_msgs,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif policy == HIDDEN_CONTENT_FULL:
|
||||||
blocks.append(
|
blocks.append(
|
||||||
make_tool_use_block(
|
make_tool_use_block(
|
||||||
item.get("name", ""), item.get("input"), item.get("id")
|
item.get("name", ""), item.get("input"), item.get("id")
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
name = item.get("name") or "tool"
|
|
||||||
size = len(json.dumps(item, default=str))
|
size = len(json.dumps(item, default=str))
|
||||||
local_tools[name] += 1
|
local_tools[name] += 1
|
||||||
local_bytes += size
|
local_bytes += size
|
||||||
|
|||||||
@@ -1,155 +0,0 @@
|
|||||||
"""Unit tests for browser cookie extraction (browser_cookie3 mocked)."""
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import types
|
|
||||||
from types import SimpleNamespace
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from src.browser_tokens import (
|
|
||||||
BrowserTokenError,
|
|
||||||
extract_chatgpt_tokens,
|
|
||||||
extract_claude_session,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _fake_bc3(monkeypatch, cookies_by_domain, raises=None):
|
|
||||||
"""Install a stub browser_cookie3 module whose ``brave`` loader returns
|
|
||||||
SimpleNamespace cookies for the requested domain (or raises)."""
|
|
||||||
module = types.ModuleType("browser_cookie3")
|
|
||||||
|
|
||||||
def brave(domain_name=""):
|
|
||||||
if raises is not None:
|
|
||||||
raise raises
|
|
||||||
return [
|
|
||||||
SimpleNamespace(name=name, value=value)
|
|
||||||
for name, value in cookies_by_domain.get(domain_name, {}).items()
|
|
||||||
]
|
|
||||||
|
|
||||||
module.brave = brave
|
|
||||||
monkeypatch.setitem(sys.modules, "browser_cookie3", module)
|
|
||||||
return module
|
|
||||||
|
|
||||||
|
|
||||||
class TestExtractChatGPT:
|
|
||||||
def test_chunked_cookies(self, monkeypatch):
|
|
||||||
_fake_bc3(monkeypatch, {
|
|
||||||
"chatgpt.com": {
|
|
||||||
"__Secure-next-auth.session-token.0": "eyJpart0",
|
|
||||||
"__Secure-next-auth.session-token.1": "part1rest",
|
|
||||||
},
|
|
||||||
})
|
|
||||||
assert extract_chatgpt_tokens("brave") == ("eyJpart0", "part1rest")
|
|
||||||
|
|
||||||
def test_single_cookie_fallback(self, monkeypatch):
|
|
||||||
_fake_bc3(monkeypatch, {
|
|
||||||
"chatgpt.com": {"__Secure-next-auth.session-token": "eyJsingle"},
|
|
||||||
})
|
|
||||||
assert extract_chatgpt_tokens("brave") == ("eyJsingle", None)
|
|
||||||
|
|
||||||
def test_missing_cookie_raises_with_login_hint(self, monkeypatch):
|
|
||||||
_fake_bc3(monkeypatch, {"chatgpt.com": {"unrelated": "x"}})
|
|
||||||
with pytest.raises(BrowserTokenError, match="log in"):
|
|
||||||
extract_chatgpt_tokens("brave")
|
|
||||||
|
|
||||||
def test_loader_failure_wrapped(self, monkeypatch):
|
|
||||||
_fake_bc3(monkeypatch, {}, raises=RuntimeError("database is locked"))
|
|
||||||
with pytest.raises(BrowserTokenError, match="Close the browser"):
|
|
||||||
extract_chatgpt_tokens("brave")
|
|
||||||
|
|
||||||
def test_unsupported_browser_rejected(self):
|
|
||||||
with pytest.raises(BrowserTokenError, match="Unsupported browser"):
|
|
||||||
extract_chatgpt_tokens("netscape")
|
|
||||||
|
|
||||||
|
|
||||||
class TestExtractClaude:
|
|
||||||
def test_session_key(self, monkeypatch):
|
|
||||||
_fake_bc3(monkeypatch, {"claude.ai": {"sessionKey": "sk-ant-sid01-xyz"}})
|
|
||||||
assert extract_claude_session("brave") == "sk-ant-sid01-xyz"
|
|
||||||
|
|
||||||
def test_missing_raises(self, monkeypatch):
|
|
||||||
_fake_bc3(monkeypatch, {"claude.ai": {}})
|
|
||||||
with pytest.raises(BrowserTokenError, match="log in"):
|
|
||||||
extract_claude_session("brave")
|
|
||||||
|
|
||||||
|
|
||||||
class TestAuthFromBrowserCLI:
|
|
||||||
def test_writes_env_after_validation(self, monkeypatch, tmp_path):
|
|
||||||
"""--from-browser extracts, validates live (mocked), and writes .env."""
|
|
||||||
from click.testing import CliRunner
|
|
||||||
from src.main import cli
|
|
||||||
from src.cache import Cache
|
|
||||||
|
|
||||||
cache = Cache(tmp_path / "cache")
|
|
||||||
cache.acknowledge_tos()
|
|
||||||
|
|
||||||
_fake_bc3(monkeypatch, {
|
|
||||||
"chatgpt.com": {
|
|
||||||
"__Secure-next-auth.session-token.0": "eyJtok0",
|
|
||||||
"__Secure-next-auth.session-token.1": "tok1",
|
|
||||||
},
|
|
||||||
"claude.ai": {"sessionKey": "sk-ant-sid01-abc"},
|
|
||||||
})
|
|
||||||
|
|
||||||
# Stub out the live validation calls.
|
|
||||||
import src.providers.chatgpt as chatgpt_mod
|
|
||||||
import src.providers.claude as claude_mod
|
|
||||||
monkeypatch.setattr(
|
|
||||||
chatgpt_mod.ChatGPTProvider, "_fetch_access_token", lambda self: "at"
|
|
||||||
)
|
|
||||||
monkeypatch.setattr(
|
|
||||||
claude_mod.ClaudeProvider,
|
|
||||||
"list_conversations",
|
|
||||||
lambda self, offset=0, limit=100: [],
|
|
||||||
)
|
|
||||||
|
|
||||||
runner = CliRunner(mix_stderr=True)
|
|
||||||
with runner.isolated_filesystem(temp_dir=tmp_path):
|
|
||||||
result = runner.invoke(
|
|
||||||
cli,
|
|
||||||
["--no-log-file", "auth", "--from-browser"],
|
|
||||||
env={"CACHE_DIR": str(tmp_path / "cache")},
|
|
||||||
)
|
|
||||||
assert result.exit_code == 0, result.output
|
|
||||||
from pathlib import Path
|
|
||||||
env_text = Path(".env").read_text()
|
|
||||||
|
|
||||||
assert "CHATGPT_SESSION_TOKEN=eyJtok0" in env_text
|
|
||||||
assert "CHATGPT_SESSION_TOKEN_1=tok1" in env_text
|
|
||||||
assert "CLAUDE_SESSION_KEY=sk-ant-sid01-abc" in env_text
|
|
||||||
assert "2 provider(s) configured" in result.output
|
|
||||||
|
|
||||||
def test_validation_failure_leaves_env_untouched(self, monkeypatch, tmp_path):
|
|
||||||
from click.testing import CliRunner
|
|
||||||
from src.main import cli
|
|
||||||
from src.cache import Cache
|
|
||||||
from src.providers.base import ProviderError
|
|
||||||
|
|
||||||
cache = Cache(tmp_path / "cache")
|
|
||||||
cache.acknowledge_tos()
|
|
||||||
|
|
||||||
_fake_bc3(monkeypatch, {
|
|
||||||
"chatgpt.com": {"__Secure-next-auth.session-token.0": "eyJstale"},
|
|
||||||
"claude.ai": {},
|
|
||||||
})
|
|
||||||
|
|
||||||
import src.providers.chatgpt as chatgpt_mod
|
|
||||||
|
|
||||||
def _fail(self):
|
|
||||||
raise ProviderError("chatgpt", "auth", RuntimeError("401"))
|
|
||||||
|
|
||||||
monkeypatch.setattr(chatgpt_mod.ChatGPTProvider, "_fetch_access_token", _fail)
|
|
||||||
|
|
||||||
runner = CliRunner(mix_stderr=True)
|
|
||||||
with runner.isolated_filesystem(temp_dir=tmp_path):
|
|
||||||
result = runner.invoke(
|
|
||||||
cli,
|
|
||||||
["--no-log-file", "auth", "--from-browser", "brave"],
|
|
||||||
env={"CACHE_DIR": str(tmp_path / "cache")},
|
|
||||||
)
|
|
||||||
import pathlib
|
|
||||||
env_exists = pathlib.Path(".env").exists()
|
|
||||||
|
|
||||||
assert result.exit_code == 1
|
|
||||||
assert "failed live validation" in result.output
|
|
||||||
assert not env_exists
|
|
||||||
@@ -151,3 +151,106 @@ class TestGetNewOrUpdated:
|
|||||||
convs = [{"id": "a", "updated_at": "2024-06-01T00:00:00Z"}]
|
convs = [{"id": "a", "updated_at": "2024-06-01T00:00:00Z"}]
|
||||||
result = tmp_cache.get_new_or_updated("claude", convs)
|
result = tmp_cache.get_new_or_updated("claude", convs)
|
||||||
assert len(result) == 1
|
assert len(result) == 1
|
||||||
|
|
||||||
|
def test_force_returns_all_cached_and_unchanged(self, tmp_cache):
|
||||||
|
tmp_cache.mark_exported("claude", "a", {"updated_at": "2024-01-01T00:00:00Z"})
|
||||||
|
convs = [
|
||||||
|
{"id": "a", "updated_at": "2024-01-01T00:00:00Z"}, # cached, unchanged
|
||||||
|
{"id": "b", "updated_at": "2024-01-02T00:00:00Z"}, # new
|
||||||
|
]
|
||||||
|
assert len(tmp_cache.get_new_or_updated("claude", convs)) == 1
|
||||||
|
assert len(tmp_cache.get_new_or_updated("claude", convs, force=True)) == 2
|
||||||
|
|
||||||
|
def test_force_orders_least_recently_exported_first(self, tmp_cache):
|
||||||
|
# 'a' was exported long ago; 'b' just now; 'c' never.
|
||||||
|
tmp_cache.mark_exported("claude", "a", {"updated_at": "2024-01-01T00:00:00Z"})
|
||||||
|
tmp_cache._data["claude"]["a"]["exported_at"] = "2024-01-01T00:00:00Z"
|
||||||
|
tmp_cache.mark_exported("claude", "b", {"updated_at": "2024-01-01T00:00:00Z"})
|
||||||
|
tmp_cache._data["claude"]["b"]["exported_at"] = "2026-06-12T00:00:00Z"
|
||||||
|
convs = [{"id": "a"}, {"id": "b"}, {"id": "c"}]
|
||||||
|
ordered = [c["id"] for c in tmp_cache.get_new_or_updated("claude", convs, force=True)]
|
||||||
|
# never-exported ('c', exported_at "") and oldest ('a') come before 'b'
|
||||||
|
assert ordered == ["c", "a", "b"]
|
||||||
|
|
||||||
|
def test_force_capped_run_makes_progress(self, tmp_cache):
|
||||||
|
"""The bug: force + cap repeated the same head every run. Now each
|
||||||
|
capped run re-exports the next-oldest batch and converges."""
|
||||||
|
convs = [{"id": f"c{i}", "updated_at": "2024-01-01T00:00:00Z"} for i in range(5)]
|
||||||
|
seen = set()
|
||||||
|
for _ in range(3): # cap=2 over 3 runs should cover all 5
|
||||||
|
batch = tmp_cache.get_new_or_updated("claude", convs, force=True)[:2]
|
||||||
|
for c in batch:
|
||||||
|
seen.add(c["id"])
|
||||||
|
tmp_cache.mark_exported("claude", c["id"], {"file_path": f"/{c['id']}.md"})
|
||||||
|
assert seen == {f"c{i}" for i in range(5)}
|
||||||
|
|
||||||
|
def test_campaign_excludes_already_refreshed(self, tmp_cache):
|
||||||
|
"""With a campaign stamp, conversations re-exported during the campaign
|
||||||
|
(exported_at >= stamp) drop out, so the remaining count shrinks to 0."""
|
||||||
|
convs = [{"id": f"c{i}"} for i in range(5)]
|
||||||
|
campaign = "2026-06-12T00:00:00+00:00"
|
||||||
|
# Nothing exported yet → all 5 are candidates.
|
||||||
|
assert len(tmp_cache.get_new_or_updated("claude", convs, force=True,
|
||||||
|
campaign_at=campaign)) == 5
|
||||||
|
# Re-export 2 "during" the campaign (after the stamp).
|
||||||
|
for cid in ("c0", "c1"):
|
||||||
|
tmp_cache.mark_exported("claude", cid, {"file_path": f"/{cid}.md"})
|
||||||
|
tmp_cache._data["claude"][cid]["exported_at"] = "2026-06-12T01:00:00+00:00"
|
||||||
|
remaining = tmp_cache.get_new_or_updated("claude", convs, force=True,
|
||||||
|
campaign_at=campaign)
|
||||||
|
assert {c["id"] for c in remaining} == {"c2", "c3", "c4"}
|
||||||
|
# Finish them → none left.
|
||||||
|
for cid in ("c2", "c3", "c4"):
|
||||||
|
tmp_cache.mark_exported("claude", cid, {"file_path": f"/{cid}.md"})
|
||||||
|
tmp_cache._data["claude"][cid]["exported_at"] = "2026-06-12T02:00:00+00:00"
|
||||||
|
assert tmp_cache.get_new_or_updated("claude", convs, force=True,
|
||||||
|
campaign_at=campaign) == []
|
||||||
|
|
||||||
|
def test_force_campaign_marker_roundtrip(self, tmp_cache):
|
||||||
|
assert tmp_cache.get_force_campaign() is None
|
||||||
|
tmp_cache.set_force_campaign("2026-06-12T00:00:00+00:00")
|
||||||
|
assert tmp_cache.get_force_campaign() == "2026-06-12T00:00:00+00:00"
|
||||||
|
# Survives reload, and is not mistaken for a provider by stats().
|
||||||
|
reopened = Cache(tmp_cache._dir)
|
||||||
|
assert reopened.get_force_campaign() == "2026-06-12T00:00:00+00:00"
|
||||||
|
assert "force_campaign_at" not in reopened.stats()
|
||||||
|
tmp_cache.clear_force_campaign()
|
||||||
|
assert tmp_cache.get_force_campaign() is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestReexportPreservesJoplinLink:
|
||||||
|
"""A re-export must not drop the Joplin note link, or re-syncing would
|
||||||
|
create duplicate notes instead of updating the existing ones."""
|
||||||
|
|
||||||
|
def test_joplin_fields_carried_across_reexport(self, tmp_cache):
|
||||||
|
tmp_cache.mark_exported("chatgpt", "c1", {
|
||||||
|
"title": "T", "updated_at": "2024-01-01T00:00:00Z", "file_path": "/x.md",
|
||||||
|
})
|
||||||
|
tmp_cache.mark_joplin_synced("chatgpt", "c1", "note-123")
|
||||||
|
tmp_cache.set_joplin_resources("chatgpt", "c1", {"media/a.png": "res-1"})
|
||||||
|
|
||||||
|
# Re-export the same conversation (new render, new file_path).
|
||||||
|
tmp_cache.mark_exported("chatgpt", "c1", {
|
||||||
|
"title": "T", "updated_at": "2024-01-01T00:00:00Z", "file_path": "/x2.md",
|
||||||
|
})
|
||||||
|
|
||||||
|
entry = tmp_cache.get_all_entries("chatgpt")["c1"]
|
||||||
|
assert entry["joplin_note_id"] == "note-123"
|
||||||
|
assert entry["joplin_resources"] == {"media/a.png": "res-1"}
|
||||||
|
assert entry["file_path"] == "/x2.md"
|
||||||
|
|
||||||
|
def test_reexport_marks_note_for_update_not_create(self, tmp_cache):
|
||||||
|
tmp_cache.mark_exported("chatgpt", "c1", {
|
||||||
|
"updated_at": "2024-01-01T00:00:00Z", "file_path": "/x.md",
|
||||||
|
})
|
||||||
|
tmp_cache.mark_joplin_synced("chatgpt", "c1", "note-123")
|
||||||
|
# Nothing pending right after sync.
|
||||||
|
assert tmp_cache.get_joplin_pending("chatgpt") == []
|
||||||
|
# Re-export bumps exported_at past joplin_synced_at → pending for update,
|
||||||
|
# carrying the existing note id so the sync updates rather than creates.
|
||||||
|
tmp_cache.mark_exported("chatgpt", "c1", {
|
||||||
|
"updated_at": "2024-01-01T00:00:00Z", "file_path": "/x.md",
|
||||||
|
})
|
||||||
|
pending = tmp_cache.get_joplin_pending("chatgpt")
|
||||||
|
assert len(pending) == 1
|
||||||
|
assert pending[0][1]["joplin_note_id"] == "note-123"
|
||||||
|
|||||||
@@ -228,3 +228,276 @@ class TestClaudeCodeProvider:
|
|||||||
rendered = render_blocks_to_markdown(result["messages"][1]["blocks"])
|
rendered = render_blocks_to_markdown(result["messages"][1]["blocks"])
|
||||||
assert "> 🔧 **Tool output** — `3 calls: Read ×2, Bash ×1`" in rendered
|
assert "> 🔧 **Tool output** — `3 calls: Read ×2, Bash ×1`" in rendered
|
||||||
assert "omitted (EXPORTER_HIDDEN_CONTENT=full to keep)" in rendered
|
assert "omitted (EXPORTER_HIDDEN_CONTENT=full to keep)" in rendered
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Subagent folding (Part B)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _write_subagent(
|
||||||
|
session_file, tool_use_id, records, agent_type="Explore",
|
||||||
|
description="do a thing", name="agent-x",
|
||||||
|
):
|
||||||
|
"""Write a <session>/subagents/<name>.jsonl + .meta.json sidecar."""
|
||||||
|
subdir = session_file.parent / session_file.stem / "subagents"
|
||||||
|
subdir.mkdir(parents=True, exist_ok=True)
|
||||||
|
(subdir / f"{name}.jsonl").write_text(
|
||||||
|
"\n".join(json.dumps(r) for r in records), encoding="utf-8"
|
||||||
|
)
|
||||||
|
(subdir / f"{name}.meta.json").write_text(
|
||||||
|
json.dumps({
|
||||||
|
"agentType": agent_type,
|
||||||
|
"description": description,
|
||||||
|
"toolUseId": tool_use_id,
|
||||||
|
}),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _parent_with_task(tool_use_id="toolu_sub1"):
|
||||||
|
return [
|
||||||
|
{"type": "ai-title", "aiTitle": "Parent session"},
|
||||||
|
{
|
||||||
|
"type": "user", "cwd": "/home/jesse/myproj",
|
||||||
|
"timestamp": "2026-05-01T10:00:00.000Z",
|
||||||
|
"message": {"role": "user", "content": "delegate the research"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "assistant", "timestamp": "2026-05-01T10:00:05.000Z",
|
||||||
|
"message": {"role": "assistant", "content": [
|
||||||
|
{"type": "text", "text": "I'll delegate this."},
|
||||||
|
{"type": "tool_use", "id": tool_use_id, "name": "Task",
|
||||||
|
"input": {"description": "research"}},
|
||||||
|
]},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _subagent_records():
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"type": "user", "isSidechain": True,
|
||||||
|
"timestamp": "2026-05-01T10:00:06.000Z",
|
||||||
|
"message": {"role": "user", "content": "Go research X"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "assistant", "isSidechain": True,
|
||||||
|
"timestamp": "2026-05-01T10:00:07.000Z",
|
||||||
|
"message": {"role": "assistant", "content": [
|
||||||
|
{"type": "text", "text": "Here is the research result."},
|
||||||
|
{"type": "tool_use", "id": "st1", "name": "Grep",
|
||||||
|
"input": {"pattern": "x"}},
|
||||||
|
]},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class TestSubagentFold:
|
||||||
|
def _provider(self, tmp_path, policy="placeholder"):
|
||||||
|
return ClaudeCodeProvider(projects_dir=tmp_path, hidden_content=policy)
|
||||||
|
|
||||||
|
def _normalize(self, tmp_path, policy="placeholder"):
|
||||||
|
f = _write_session(tmp_path, _parent_with_task(), name="parent-1")
|
||||||
|
_write_subagent(
|
||||||
|
f, "toolu_sub1", _subagent_records(),
|
||||||
|
agent_type="Explore", description="research X",
|
||||||
|
)
|
||||||
|
p = self._provider(tmp_path, policy)
|
||||||
|
p.fetch_all_conversations()
|
||||||
|
return p.normalize_conversation(p.get_conversation("parent-1"))
|
||||||
|
|
||||||
|
def test_subagent_folded_into_parent(self, tmp_path):
|
||||||
|
conv = self._normalize(tmp_path)
|
||||||
|
blocks = [b for m in conv["messages"] for b in m["blocks"]]
|
||||||
|
subs = [b for b in blocks if b["type"] == "subagent"]
|
||||||
|
assert len(subs) == 1
|
||||||
|
assert subs[0]["agent_type"] == "Explore"
|
||||||
|
assert subs[0]["description"] == "research X"
|
||||||
|
inner_text = [
|
||||||
|
bb.get("text") for m in subs[0]["messages"] for bb in m["blocks"]
|
||||||
|
]
|
||||||
|
assert "Go research X" in inner_text
|
||||||
|
assert "Here is the research result." in inner_text
|
||||||
|
|
||||||
|
def test_subagent_tools_collapsed(self, tmp_path):
|
||||||
|
conv = self._normalize(tmp_path)
|
||||||
|
sub = next(
|
||||||
|
b for m in conv["messages"] for b in m["blocks"]
|
||||||
|
if b["type"] == "subagent"
|
||||||
|
)
|
||||||
|
inner_blocks = [bb for m in sub["messages"] for bb in m["blocks"]]
|
||||||
|
assert any(b["type"] == BLOCK_TYPE_COLLAPSED for b in inner_blocks)
|
||||||
|
assert not any(b["type"] == BLOCK_TYPE_TOOL_USE for b in inner_blocks)
|
||||||
|
collapsed = next(b for b in inner_blocks if b["type"] == BLOCK_TYPE_COLLAPSED)
|
||||||
|
assert "Grep" in collapsed["origin"]
|
||||||
|
|
||||||
|
def test_subagent_not_listed_as_conversation(self, tmp_path):
|
||||||
|
f = _write_session(tmp_path, _parent_with_task(), name="parent-1")
|
||||||
|
_write_subagent(f, "toolu_sub1", _subagent_records())
|
||||||
|
p = self._provider(tmp_path)
|
||||||
|
convs = p.fetch_all_conversations()
|
||||||
|
assert [c["id"] for c in convs] == ["parent-1"]
|
||||||
|
|
||||||
|
def test_renders_as_details_block(self, tmp_path):
|
||||||
|
conv = self._normalize(tmp_path)
|
||||||
|
spawning = conv["messages"][1] # the assistant turn with the Task call
|
||||||
|
rendered = render_blocks_to_markdown(spawning["blocks"])
|
||||||
|
assert "<details>" in rendered
|
||||||
|
assert "<summary>🤖 Subagent: Explore — research X</summary>" in rendered
|
||||||
|
assert "</details>" in rendered
|
||||||
|
assert "Here is the research result." in rendered
|
||||||
|
|
||||||
|
def test_main_pass_still_skips_stray_sidechain(self, tmp_path):
|
||||||
|
# A sidechain record with no matching subagent file must NOT leak into
|
||||||
|
# the top-level dialogue.
|
||||||
|
records = _parent_with_task() + [
|
||||||
|
{"type": "assistant", "isSidechain": True,
|
||||||
|
"message": {"role": "assistant",
|
||||||
|
"content": [{"type": "text", "text": "stray sidechain"}]}},
|
||||||
|
]
|
||||||
|
f = _write_session(tmp_path, records, name="parent-2")
|
||||||
|
_write_subagent(f, "toolu_sub1", _subagent_records())
|
||||||
|
p = self._provider(tmp_path)
|
||||||
|
p.fetch_all_conversations()
|
||||||
|
conv = p.normalize_conversation(p.get_conversation("parent-2"))
|
||||||
|
top_text = [
|
||||||
|
b.get("text") for m in conv["messages"] for b in m["blocks"]
|
||||||
|
]
|
||||||
|
assert "stray sidechain" not in top_text
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Repo tags in title (Part C)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestRepoTags:
|
||||||
|
@staticmethod
|
||||||
|
def _mkrepo(base, name):
|
||||||
|
"""Create a git repo dir (bare .git marker) and return its path."""
|
||||||
|
(base / name / ".git").mkdir(parents=True, exist_ok=True)
|
||||||
|
return base / name
|
||||||
|
|
||||||
|
def _title(self, tmp_path, content):
|
||||||
|
(tmp_path / "ws").mkdir(exist_ok=True)
|
||||||
|
records = [
|
||||||
|
{"type": "ai-title", "aiTitle": "Work session"},
|
||||||
|
{"type": "user", "cwd": str(tmp_path / "ws"),
|
||||||
|
"timestamp": "2026-05-01T10:00:00.000Z",
|
||||||
|
"message": {"role": "user", "content": "go"}},
|
||||||
|
{"type": "assistant",
|
||||||
|
"message": {"role": "assistant", "content": content}},
|
||||||
|
]
|
||||||
|
_write_session(tmp_path, records, name="ws-1")
|
||||||
|
p = ClaudeCodeProvider(projects_dir=tmp_path)
|
||||||
|
p.fetch_all_conversations()
|
||||||
|
return p.normalize_conversation(p.get_conversation("ws-1"))["title"]
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _read(path):
|
||||||
|
return {"type": "tool_use", "id": str(path), "name": "Read",
|
||||||
|
"input": {"file_path": str(path)}}
|
||||||
|
|
||||||
|
def test_repos_by_git_root_ordered_by_frequency(self, tmp_path):
|
||||||
|
a = self._mkrepo(tmp_path, "repo-a")
|
||||||
|
b = self._mkrepo(tmp_path, "repo-b")
|
||||||
|
content = [
|
||||||
|
self._read(a / "src/x.py"), self._read(a / "src/y.py"),
|
||||||
|
self._read(b / "z.py"),
|
||||||
|
# a file not inside any git repo → excluded
|
||||||
|
self._read(tmp_path / "CLAUDE.md"),
|
||||||
|
]
|
||||||
|
assert self._title(tmp_path, content) == "Work session [repo-a, repo-b]"
|
||||||
|
|
||||||
|
def test_cross_workspace_repo_tagged(self, tmp_path):
|
||||||
|
# Launched in ws/, but touched a repo in a sibling tree — still tagged.
|
||||||
|
other = self._mkrepo(tmp_path / "elsewhere", "faraway-repo")
|
||||||
|
content = [self._read(other / "deep/nested/file.py")]
|
||||||
|
assert self._title(tmp_path, content) == "Work session [faraway-repo]"
|
||||||
|
|
||||||
|
def test_no_repos_no_bracket(self, tmp_path):
|
||||||
|
content = [self._read(tmp_path / "loose/file.txt")] # no .git anywhere
|
||||||
|
assert self._title(tmp_path, content) == "Work session"
|
||||||
|
|
||||||
|
def test_cap_three_with_ellipsis(self, tmp_path):
|
||||||
|
content = [
|
||||||
|
self._read(self._mkrepo(tmp_path, f"repo-{c}") / "f/x.py")
|
||||||
|
for c in "abcd"
|
||||||
|
]
|
||||||
|
title = self._title(tmp_path, content)
|
||||||
|
assert title.startswith("Work session [repo-a, repo-b, repo-c, …]")
|
||||||
|
assert "repo-d" not in title
|
||||||
|
|
||||||
|
def test_paths_inside_bash_commands_count(self, tmp_path):
|
||||||
|
x = self._mkrepo(tmp_path, "repo-x")
|
||||||
|
content = [
|
||||||
|
{"type": "tool_use", "id": "a", "name": "Bash",
|
||||||
|
"input": {"command": f"cat {x / 'main.go'}"}},
|
||||||
|
]
|
||||||
|
assert self._title(tmp_path, content) == "Work session [repo-x]"
|
||||||
|
|
||||||
|
def test_ignore_list_via_env(self, tmp_path, monkeypatch):
|
||||||
|
a = self._mkrepo(tmp_path, "repo-a")
|
||||||
|
b = self._mkrepo(tmp_path, "repo-b")
|
||||||
|
monkeypatch.setenv("CLAUDE_CODE_REPO_TAG_IGNORE", "repo-b, notes")
|
||||||
|
content = [self._read(a / "x.py"), self._read(b / "y.py")]
|
||||||
|
assert self._title(tmp_path, content) == "Work session [repo-a]"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Multiple projects roots (Part D)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestMultiRoot:
|
||||||
|
def _titled(self, aititle, cwd="/home/jesse/p"):
|
||||||
|
return [
|
||||||
|
{"type": "ai-title", "aiTitle": aititle},
|
||||||
|
{"type": "user", "cwd": cwd, "timestamp": "2026-05-01T10:00:00.000Z",
|
||||||
|
"message": {"role": "user", "content": "hi"}},
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_two_roots_merged(self, tmp_path):
|
||||||
|
r1, r2 = tmp_path / "root1", tmp_path / "root2"
|
||||||
|
_write_session(r1, self._titled("From root1"), name="s1")
|
||||||
|
_write_session(r2, self._titled("From root2"), name="s2")
|
||||||
|
p = ClaudeCodeProvider(projects_dir=[r1, r2])
|
||||||
|
ids = sorted(c["id"] for c in p.fetch_all_conversations())
|
||||||
|
assert ids == ["s1", "s2"]
|
||||||
|
|
||||||
|
def test_duplicate_uuid_newer_wins(self, tmp_path):
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
r1, r2 = tmp_path / "root1", tmp_path / "root2"
|
||||||
|
_write_session(r1, self._titled("Old copy"), name="dup")
|
||||||
|
f2 = _write_session(r2, self._titled("New copy"), name="dup")
|
||||||
|
os.utime(f2, (time.time() + 10, time.time() + 10))
|
||||||
|
p = ClaudeCodeProvider(projects_dir=[r1, r2])
|
||||||
|
convs = p.fetch_all_conversations()
|
||||||
|
assert len(convs) == 1
|
||||||
|
assert convs[0]["title"] == "New copy"
|
||||||
|
assert p._path_map["dup"] == f2
|
||||||
|
|
||||||
|
def test_single_path_backward_compatible(self, tmp_path):
|
||||||
|
_write_session(tmp_path, _records())
|
||||||
|
p = ClaudeCodeProvider(projects_dir=tmp_path)
|
||||||
|
assert len(p.fetch_all_conversations()) == 1
|
||||||
|
|
||||||
|
def test_env_pathsep_list(self, tmp_path, monkeypatch):
|
||||||
|
import os
|
||||||
|
from src.providers.claude_code import resolve_roots
|
||||||
|
r1, r2 = tmp_path / "root1", tmp_path / "root2"
|
||||||
|
_write_session(r1, self._titled("From root1"), name="s1")
|
||||||
|
_write_session(r2, self._titled("From root2"), name="s2")
|
||||||
|
monkeypatch.setenv("CLAUDE_CODE_DIR", f"{r1}{os.pathsep}{r2}")
|
||||||
|
assert r1 in resolve_roots() and r2 in resolve_roots()
|
||||||
|
p = ClaudeCodeProvider()
|
||||||
|
assert len(p.fetch_all_conversations()) == 2
|
||||||
|
|
||||||
|
def test_config_dir_projects_included(self, tmp_path, monkeypatch):
|
||||||
|
from src.providers.claude_code import resolve_roots
|
||||||
|
monkeypatch.delenv("CLAUDE_CODE_DIR", raising=False)
|
||||||
|
monkeypatch.setenv("CLAUDE_CONFIG_DIR", str(tmp_path / "cfg"))
|
||||||
|
roots = resolve_roots()
|
||||||
|
assert (tmp_path / "cfg" / "projects") in roots
|
||||||
|
|||||||
@@ -275,3 +275,25 @@ class TestPrune:
|
|||||||
assert result.exit_code == 0
|
assert result.exit_code == 0
|
||||||
assert "Aborted" in result.output
|
assert "Aborted" in result.output
|
||||||
assert stale.exists()
|
assert stale.exists()
|
||||||
|
|
||||||
|
|
||||||
|
class TestCanaryCommand:
|
||||||
|
"""`canary` wiring — no live API calls (no tokens configured)."""
|
||||||
|
|
||||||
|
def test_no_tokens_exits_nonzero(self, tmp_path):
|
||||||
|
cache = Cache(tmp_path / "cache")
|
||||||
|
cache.acknowledge_tos()
|
||||||
|
runner = CliRunner(mix_stderr=True)
|
||||||
|
result = runner.invoke(
|
||||||
|
cli,
|
||||||
|
["--no-log-file", "canary"],
|
||||||
|
env={
|
||||||
|
"CACHE_DIR": str(tmp_path / "cache"),
|
||||||
|
"EXPORT_DIR": str(tmp_path / "exports"),
|
||||||
|
# Empty so .env (override=False) can't repopulate real tokens.
|
||||||
|
"CHATGPT_SESSION_TOKEN": "",
|
||||||
|
"CLAUDE_SESSION_KEY": "",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
assert result.exit_code == 1
|
||||||
|
assert "No web-API provider tokens" in result.output
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ from src.blocks import (
|
|||||||
make_file_placeholder,
|
make_file_placeholder,
|
||||||
make_hidden_context_marker,
|
make_hidden_context_marker,
|
||||||
make_image_placeholder,
|
make_image_placeholder,
|
||||||
|
make_subagent_block,
|
||||||
make_text_block,
|
make_text_block,
|
||||||
make_thinking_block,
|
make_thinking_block,
|
||||||
make_tool_result_block,
|
make_tool_result_block,
|
||||||
@@ -119,6 +120,30 @@ class TestMarkdownFrontmatter:
|
|||||||
assert "```python" in content
|
assert "```python" in content
|
||||||
assert "print('hello')" in content
|
assert "print('hello')" in content
|
||||||
|
|
||||||
|
def test_subagent_block_renders_as_details(self, tmp_path):
|
||||||
|
sub = make_subagent_block(
|
||||||
|
agent_type="Explore",
|
||||||
|
description="find the thing",
|
||||||
|
messages=[
|
||||||
|
{"role": "user", "blocks": [make_text_block("Go find it")]},
|
||||||
|
{"role": "assistant", "blocks": [make_text_block("Found it here.")]},
|
||||||
|
],
|
||||||
|
)
|
||||||
|
conv = {
|
||||||
|
**SAMPLE_CONV,
|
||||||
|
"provider": "claude-code",
|
||||||
|
"messages": [
|
||||||
|
{"role": "assistant", "content_type": "text", "timestamp": None,
|
||||||
|
"blocks": [make_text_block("Delegating."), sub]},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
content = MarkdownExporter(tmp_path).export(conv).read_text()
|
||||||
|
assert "<details>" in content
|
||||||
|
assert "<summary>🤖 Subagent: Explore — find the thing</summary>" in content
|
||||||
|
assert "Go find it" in content
|
||||||
|
assert "Found it here." in content
|
||||||
|
assert "</details>" in content
|
||||||
|
|
||||||
|
|
||||||
class TestMarkdownFilenameGeneration:
|
class TestMarkdownFilenameGeneration:
|
||||||
def test_filename_format(self, tmp_path):
|
def test_filename_format(self, tmp_path):
|
||||||
|
|||||||
@@ -48,6 +48,9 @@ class TestNotebookPath:
|
|||||||
def test_claude_provider(self):
|
def test_claude_provider(self):
|
||||||
assert notebook_path("claude", "budget-tracker") == ("AI-Claude", "Budget Tracker")
|
assert notebook_path("claude", "budget-tracker") == ("AI-Claude", "Budget Tracker")
|
||||||
|
|
||||||
|
def test_claude_code_gets_own_top_level_notebook(self):
|
||||||
|
assert notebook_path("claude-code", "services") == ("AI-ClaudeCode", "Services")
|
||||||
|
|
||||||
def test_multi_word_project(self):
|
def test_multi_word_project(self):
|
||||||
assert notebook_path("claude", "ai-research-notes") == ("AI-Claude", "Ai Research Notes")
|
assert notebook_path("claude", "ai-research-notes") == ("AI-Claude", "Ai Research Notes")
|
||||||
|
|
||||||
@@ -61,6 +64,24 @@ class TestNotebookPath:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestUpdateNote:
|
||||||
|
def test_parent_id_moves_note(self):
|
||||||
|
client = _make_client()
|
||||||
|
client._put = MagicMock()
|
||||||
|
client.update_note("nid", "Title", "body", parent_id="nb1")
|
||||||
|
args, _ = client._put.call_args
|
||||||
|
assert args[0] == "/notes/nid"
|
||||||
|
assert args[1]["parent_id"] == "nb1"
|
||||||
|
assert args[1]["title"] == "Title"
|
||||||
|
|
||||||
|
def test_no_parent_id_omits_field(self):
|
||||||
|
client = _make_client()
|
||||||
|
client._put = MagicMock()
|
||||||
|
client.update_note("nid", "Title", "body")
|
||||||
|
args, _ = client._put.call_args
|
||||||
|
assert "parent_id" not in args[1]
|
||||||
|
|
||||||
|
|
||||||
class TestPing:
|
class TestPing:
|
||||||
def test_ping_success(self):
|
def test_ping_success(self):
|
||||||
client = _make_client()
|
client = _make_client()
|
||||||
|
|||||||
@@ -873,3 +873,246 @@ class TestChatGPTConvIdFallback:
|
|||||||
p = self._get_provider()
|
p = self._get_provider()
|
||||||
result = p.normalize_conversation(raw)
|
result = p.normalize_conversation(raw)
|
||||||
assert result["id"] == "from-id"
|
assert result["id"] == "from-id"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# ChatGPT session-token health (§9): /api/auth/session `error` is the signal,
|
||||||
|
# not `expires`/`accessToken`. Verified live 2026-06-28 — a dead token returns
|
||||||
|
# HTTP 200 with error=RefreshAccessTokenError and a stale accessToken.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeResp:
|
||||||
|
def __init__(self, payload, status=200):
|
||||||
|
self._payload = payload
|
||||||
|
self.status_code = status
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
if self.status_code >= 400:
|
||||||
|
raise RuntimeError(f"HTTP {self.status_code}")
|
||||||
|
|
||||||
|
def json(self):
|
||||||
|
return self._payload
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeSession:
|
||||||
|
"""Minimal stand-in for the provider's HTTP session."""
|
||||||
|
|
||||||
|
def __init__(self, payload=None, status=200, raises=None):
|
||||||
|
self._payload = payload
|
||||||
|
self._status = status
|
||||||
|
self._raises = raises
|
||||||
|
|
||||||
|
def get(self, *args, **kwargs):
|
||||||
|
if self._raises is not None:
|
||||||
|
raise self._raises
|
||||||
|
return _FakeResp(self._payload, self._status)
|
||||||
|
|
||||||
|
|
||||||
|
class TestChatGPTSessionHealth:
|
||||||
|
def _provider(self, session):
|
||||||
|
from src.providers.chatgpt import ChatGPTProvider
|
||||||
|
|
||||||
|
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||||
|
p._session = session
|
||||||
|
return p
|
||||||
|
|
||||||
|
def test_fetch_access_token_returns_token_when_healthy(self):
|
||||||
|
p = self._provider(_FakeSession({"accessToken": "tok-123", "error": None}))
|
||||||
|
assert p._fetch_access_token() == "tok-123"
|
||||||
|
|
||||||
|
def test_fetch_access_token_fails_fast_on_refresh_error(self):
|
||||||
|
from src.providers.base import ProviderError
|
||||||
|
|
||||||
|
# Dead token: HTTP 200, error set, stale accessToken still present.
|
||||||
|
p = self._provider(
|
||||||
|
_FakeSession({"accessToken": "stale", "error": "RefreshAccessTokenError"})
|
||||||
|
)
|
||||||
|
with pytest.raises(ProviderError) as exc:
|
||||||
|
p._fetch_access_token()
|
||||||
|
assert "expired" in str(exc.value.original).lower()
|
||||||
|
|
||||||
|
def test_fetch_access_token_fails_when_no_token(self):
|
||||||
|
from src.providers.base import ProviderError
|
||||||
|
|
||||||
|
p = self._provider(_FakeSession({"error": None}))
|
||||||
|
with pytest.raises(ProviderError):
|
||||||
|
p._fetch_access_token()
|
||||||
|
|
||||||
|
def test_session_health_ok_when_no_error(self):
|
||||||
|
p = self._provider(_FakeSession({"accessToken": "tok", "error": None}))
|
||||||
|
ok, detail = p.session_health()
|
||||||
|
assert ok is True
|
||||||
|
assert detail == "Session active"
|
||||||
|
|
||||||
|
def test_session_health_reports_expired_on_error(self):
|
||||||
|
p = self._provider(
|
||||||
|
_FakeSession({"accessToken": "stale", "error": "RefreshAccessTokenError"})
|
||||||
|
)
|
||||||
|
ok, detail = p.session_health()
|
||||||
|
assert ok is False
|
||||||
|
assert "RefreshAccessTokenError" in detail
|
||||||
|
assert "refresh" in detail.lower()
|
||||||
|
|
||||||
|
def test_session_health_does_not_use_rolling_expires(self):
|
||||||
|
# A far-future `expires` must NOT make a dead token look healthy.
|
||||||
|
p = self._provider(
|
||||||
|
_FakeSession(
|
||||||
|
{
|
||||||
|
"accessToken": "stale",
|
||||||
|
"error": "RefreshAccessTokenError",
|
||||||
|
"expires": "2026-09-26T05:07:31.108Z",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
ok, _ = p.session_health()
|
||||||
|
assert ok is False
|
||||||
|
|
||||||
|
def test_session_health_graceful_on_network_error(self):
|
||||||
|
p = self._provider(_FakeSession(raises=RuntimeError("boom")))
|
||||||
|
ok, detail = p.session_health()
|
||||||
|
assert ok is False
|
||||||
|
assert "unreachable" in detail.lower()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# API-drift canary (§10): assert load-bearing fields, flag silent risks.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _sev(findings):
|
||||||
|
return {f["severity"] for f in findings}
|
||||||
|
|
||||||
|
|
||||||
|
def _checks(findings, severity):
|
||||||
|
return {f["check"] for f in findings if f["severity"] == severity}
|
||||||
|
|
||||||
|
|
||||||
|
class TestChatGPTDriftCanary:
|
||||||
|
def _provider(self, page, detail):
|
||||||
|
from src.providers.chatgpt import ChatGPTProvider
|
||||||
|
|
||||||
|
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||||
|
p.list_conversations = lambda offset=0, limit=5: page
|
||||||
|
p.get_conversation = lambda cid: detail
|
||||||
|
return p
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _detail(mapping):
|
||||||
|
return {
|
||||||
|
"conversation_id": "c1", "title": "T",
|
||||||
|
"create_time": 1.0, "update_time": 2.0, "mapping": mapping,
|
||||||
|
}
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _node(role="user", name=None, content_type="text", parts=("hello",)):
|
||||||
|
return {
|
||||||
|
"id": "n", "parent": None, "children": [],
|
||||||
|
"message": {
|
||||||
|
"author": {"role": role, "name": name},
|
||||||
|
"content": {"content_type": content_type, "parts": list(parts)},
|
||||||
|
"metadata": {},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_healthy_shape_is_ok_only(self):
|
||||||
|
from src.providers.base import DRIFT_OK
|
||||||
|
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail({"n1": self._node()}))
|
||||||
|
assert _sev(p.check_drift()) == {DRIFT_OK}
|
||||||
|
|
||||||
|
def test_missing_listing_id_is_error(self):
|
||||||
|
from src.providers.base import DRIFT_ERROR
|
||||||
|
|
||||||
|
p = self._provider([{"title": "T"}], self._detail({"n1": self._node()}))
|
||||||
|
assert DRIFT_ERROR in _sev(p.check_drift())
|
||||||
|
|
||||||
|
def test_empty_mapping_is_error(self):
|
||||||
|
from src.providers.base import DRIFT_ERROR
|
||||||
|
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail({}))
|
||||||
|
findings = p.check_drift()
|
||||||
|
assert "detail" in _checks(findings, DRIFT_ERROR)
|
||||||
|
|
||||||
|
def test_unfamiliar_tool_author_warns_collapse(self):
|
||||||
|
from src.providers.base import DRIFT_WARN
|
||||||
|
|
||||||
|
mapping = {
|
||||||
|
"n1": self._node(), # a healthy text msg so parts check passes
|
||||||
|
"n2": self._node(role="tool", name="brand_new_retriever"),
|
||||||
|
}
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail(mapping))
|
||||||
|
assert "collapse" in _checks(p.check_drift(), DRIFT_WARN)
|
||||||
|
|
||||||
|
def test_new_content_type_warns(self):
|
||||||
|
from src.providers.base import DRIFT_WARN
|
||||||
|
|
||||||
|
mapping = {
|
||||||
|
"n1": self._node(),
|
||||||
|
"n2": self._node(content_type="hologram_v2", parts=()),
|
||||||
|
}
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail(mapping))
|
||||||
|
assert "content_type" in _checks(p.check_drift(), DRIFT_WARN)
|
||||||
|
|
||||||
|
def test_known_tool_author_not_flagged(self):
|
||||||
|
# file_search is a known collapse author — must NOT warn.
|
||||||
|
mapping = {
|
||||||
|
"n1": self._node(),
|
||||||
|
"n2": self._node(role="tool", name="file_search"),
|
||||||
|
}
|
||||||
|
p = self._provider([{"id": "c1", "title": "T", "update_time": "x"}],
|
||||||
|
self._detail(mapping))
|
||||||
|
assert "collapse" not in _checks(p.check_drift(), "warn")
|
||||||
|
|
||||||
|
|
||||||
|
class TestClaudeDriftCanary:
|
||||||
|
def _provider(self, page, detail):
|
||||||
|
from src.providers.claude import ClaudeProvider
|
||||||
|
|
||||||
|
p = ClaudeProvider.__new__(ClaudeProvider)
|
||||||
|
p.list_conversations = lambda offset=0, limit=5: page
|
||||||
|
p.get_conversation = lambda cid: detail
|
||||||
|
return p
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _detail(msgs):
|
||||||
|
return {"uuid": "u1", "name": "N", "created_at": "x", "updated_at": "y",
|
||||||
|
"chat_messages": msgs}
|
||||||
|
|
||||||
|
def test_healthy_flat_shape_is_ok(self):
|
||||||
|
from src.providers.base import DRIFT_OK
|
||||||
|
|
||||||
|
msgs = [{"uuid": "m1", "sender": "human", "text": "hi",
|
||||||
|
"created_at": "x", "attachments": [], "files": []}]
|
||||||
|
p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
|
||||||
|
self._detail(msgs))
|
||||||
|
assert _sev(p.check_drift()) == {DRIFT_OK}
|
||||||
|
|
||||||
|
def test_content_as_list_warns(self):
|
||||||
|
from src.providers.base import DRIFT_WARN
|
||||||
|
|
||||||
|
msgs = [{"uuid": "m1", "sender": "assistant",
|
||||||
|
"content": [{"type": "text", "text": "hi"}], "created_at": "x"}]
|
||||||
|
p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
|
||||||
|
self._detail(msgs))
|
||||||
|
assert "content" in _checks(p.check_drift(), DRIFT_WARN)
|
||||||
|
|
||||||
|
def test_nonempty_attachments_warn(self):
|
||||||
|
from src.providers.base import DRIFT_WARN
|
||||||
|
|
||||||
|
msgs = [{"uuid": "m1", "sender": "human", "text": "hi",
|
||||||
|
"created_at": "x", "attachments": [{"file_name": "a.pdf"}]}]
|
||||||
|
p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
|
||||||
|
self._detail(msgs))
|
||||||
|
assert "attachments" in _checks(p.check_drift(), DRIFT_WARN)
|
||||||
|
|
||||||
|
def test_empty_messages_is_error(self):
|
||||||
|
from src.providers.base import DRIFT_ERROR
|
||||||
|
|
||||||
|
p = self._provider([{"uuid": "u1", "name": "N", "updated_at": "z"}],
|
||||||
|
self._detail([]))
|
||||||
|
assert DRIFT_ERROR in _sev(p.check_drift())
|
||||||
|
|||||||
Reference in New Issue
Block a user