AIChatExporter/CHANGELOG.md

Changelog

All notable changes to this project will be documented here. Format follows Keep a Changelog.

[Unreleased]

Fixed

• 4xx errors now report why. _make_request ended non-retryable statuses with raise_for_status(), whose curl_cffi message is HTTP Error {code}: {reason} — and HTTP/2 carries no reason phrase, so a refused request logged as bare HTTP Error 403: and the response body (the only explanation the provider gives) was discarded. The body's detail/error/message is now carried into the ProviderError, redacted and truncated. This is what made the media 403s on GET /backend-api/files/{id}/download undiagnosable.
• redact_secrets missed compound key names. It matched keys exactly, so access_token, api_key, and session-token passed through un-redacted into debug-logged response bodies; matching now applies per word ("keywords", "monkey", "tokenizer" stay intact).
• tests/test_config.py::TestSessionLimiterConfig::test_defaults depended on the developer's .env. load_config() calls load_dotenv(override=False), which re-populated the variable the test had just deleted — so it passed only on a machine with no .env. The test now stubs dotenv discovery.

Changed

• Media download failures are bucketed as forbidden (403 — the asset exists, we were refused) separately from download-error, so the run summary distinguishes it from expired-or-missing (404).

[0.8.0] - 2026-07-06

Focused on the Claude Code provider after the tool's on-disk layout changed and Claude Code sessions proved hard to find in Joplin.

Added

• Subagent capture. Claude Code now stores subagent (Task-tool) transcripts as separate <session>/subagents/agent-*.jsonl files (with an agent-*.meta.json sidecar). These were previously invisible to the exporter — all delegated work (reviews, research, plans) was lost. Each subagent is now folded into its parent session inline at the Task/Agent call that spawned it, rendered as a collapsible <details> block labeled with its agentType/description. The subagent's own tool traffic is collapsed under the same EXPORTER_HIDDEN_CONTENT policy as the main dialogue; nesting is handled at any depth via toolUseId matching.
• Repo tags in Claude Code titles. Sessions launched from a workspace root all land in one folder-named notebook, so titles now carry the repos each session touched, e.g. Resume StartWRT project work [start-technologies] — visible in the note list and matched by Joplin search. A file's repo is the git repository it lives in (nearest ancestor with a .git), resolved from tool_use paths anywhere in the filesystem — so cross-workspace work is captured and non-repo noise (config dirs, one-off files, reference dirs) is excluded because it isn't a git repo. Frequency-ordered, capped at 3. Escape hatch: CLAUDE_CODE_REPO_TAG_IGNORE (comma-separated repo names). Tags reflect the current git layout, so a since-deleted/moved repo drops from the tag on re-export.
• Multiple projects roots. CLAUDE_CODE_DIR now accepts an os.pathsep-separated list of roots (a single path stays backward compatible), and CLAUDE_CONFIG_DIR's projects/ tree is scanned automatically when set. Sessions from all roots are merged by launch-folder; a session UUID present in two roots keeps the newer-mtime copy.

Changed

• Claude Code gets its own top-level Joplin notebook, AI-ClaudeCode (was nested under AI-Claude alongside Claude web chats, which made dev sessions hard to find). Existing Claude Code notes self-heal into the new notebook on the next joplin run — update_note now sets parent_id, so a changed provider→notebook mapping relocates notes in place instead of duplicating them. After migrating, the emptied AI-Claude/{Myworkspace,Services,…} notebooks can be deleted by hand (Joplin does not auto-remove empty folders).

Notes

• The sibling <session>/tool-results/*.txt sidecars (externalized large tool outputs) are intentionally not captured — tool_result content is collapsed under the default policy anyway.

[0.7.0] - 2026-06-28

Added

• canary command — probes the ChatGPT/Claude web APIs for schema drift against the fields the parser actually depends on (one listing page + one conversation per provider), not the full response shape. Flags the silent-failure risks for a backup tool: a renamed retrieval-tool author bypassing the hidden-content collapse, a new content_type, drifted message fields, or non-empty Claude attachments/files the normalizer ignores. ERROR findings exit non-zero; WARN findings are surfaced but non-fatal. See FUTURE.md §10.
• doctor now reports a real ChatGPT token-health check ("ChatGPT token active") based on the /api/auth/session error field. ChatGPT session tokens are JWEs whose exp is encrypted and unreadable client-side; the previous decode path could never yield an expiry. The honest signal is error == "RefreshAccessTokenError" (verified live), which means the session token is dead even though expires/accessToken still echo stale values. See FUTURE.md §9.

Fixed

• ChatGPTProvider._fetch_access_token now checks the /api/auth/session error field and fails fast with a clear "refresh your token" message. Previously it returned the stale accessToken present on a dead session, producing a confusing downstream 401 instead of an actionable auth error.

Removed

• auth --from-browser and the browser-cookie3 dependency (src/browser_tokens.py). Browser cookie auto-extraction is not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave) keys cookies off a SYSTEM-level layer that can't be decrypted off disk without Administrator rights and AV-flagged SYSTEM impersonation, and fails on Brave specifically (symptom: Unable to get key for cookie decryption). Auth is manual (DevTools wizard) only. See FUTURE.md §2 for the full rationale.

[0.6.0] - 2026-06-12

First tagged release. The project was developed through several internal milestones (0.1.0–0.5.0) that were never published; their changes are consolidated here.

Added

Core export & sync

• ChatGPT and Claude export via their internal web APIs, with a local cache/manifest for incremental, resumable sync (only new or updated conversations are fetched; each is written to the manifest immediately).
• Markdown and JSON exporters. Markdown rendering happens at exporter-write time (providers produce typed blocks; exporters render them), which keeps the door open for future Obsidian/HTML output.
• CLI: export, list, cache, doctor, auth, joplin, prune.
• --project filter on export/list/joplin (case-insensitive substring, or none).
• ChatGPT Projects support via CHATGPT_PROJECT_IDS (project conversations are fetched separately from the default listing).

Joplin integration

• joplin command syncs exported Markdown to Joplin as notes; notebooks are created automatically and nested per provider+project. Re-running is safe — the Joplin note ID is stored in the manifest, so notes are updated, not duplicated.
• JOPLIN_API_TOKEN, JOPLIN_API_URL, JOPLIN_REQUEST_TIMEOUT config, with actionable error messages on timeout/connection failure.

Rich content (typed blocks)

• Messages carry an ordered blocks list: text, code, thinking, tool_use, tool_result, citation, image_placeholder, file_placeholder, unknown.
• ChatGPT voice mode: audio_transcription parts render as text; audio asset pointers render as 📎 File attached placeholders with size/duration. Custom Instructions (user_editable_context/model_editable_context) now appear (were silently dropped) with a > ℹ️ Hidden context marker.
• ChatGPT execution_output, system_error, and tether_browsing_display render as tool_result blocks (with tool_name, is_error, optional summary); transient browse spinners skip silently.
• Defensive Claude block extraction (text/thinking/tool_use/tool_result/image, including nested-block flattening).
• _safe_fence picks a backtick fence longer than any run in the content, so embedded triple-backticks can't corrupt rendering (verified live in Joplin).
• Data-loss visibility: a LossReport summary at the end of every export run breaks down unknown blocks and extraction failures by raw type; unknown blocks render as visible > ⚠️ Unsupported content with the raw type and observed keys.

Invisible-content collapse (EXPORTER_HIDDEN_CONTENT=full|placeholder|omit, default placeholder; export --hidden-content)

• Messages that were invisible in the ChatGPT web UI are collapsed to one-line placeholders instead of exported in full. Two triggers: (a) tool-role retrieval dumps identified by author.name (file_search, myfiles_browser) — ChatGPT re-injects the full text of attached/project files on every tool run, measured at 86% of all content bytes across the three largest conversations and not hidden-flagged; (b) messages flagged is_visually_hidden_from_conversation (Custom Instructions). Code-execution results, web-search output, and all dialogue are untouched.
• New collapsed block renders as > 🔧 **Tool output** — file_search (15.0 KB) — omitted (EXPORTER_HIDDEN_CONTENT=full to keep) (ℹ️ Hidden context variant for hidden-flagged messages). The LossReport gains a collapsed by policy section (count per origin + total KB) so the omission stays visible.

Claude Code session provider (--provider claude-code)

• Archives local agent transcripts from ~/.claude/projects/*/*.jsonl (override with CLAUDE_CODE_DIR) — no tokens, no rate limits, no ToS exposure. Prose-only by default per the hidden-content policy: dialogue and deliverables kept; tool traffic grouped into one placeholder per activity run (> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted); thinking dropped (counted in the summary); subagent (isSidechain) and harness (isMeta, snapshots, command tags) records stripped. Titles from the last ai-title record; project from the working-directory basename. Joplin notebooks nest under the AI-Claude parent. Measured: 19MB of JSONL → 804KB of Markdown across 25 sessions.

Binary/image downloads (EXPORTER_DOWNLOAD_MEDIA=images|all|off, default images; export --download-media)

• Downloads ChatGPT conversation assets into a media/ folder beside each export and inlines them — images become real ![](media/…) embeds, files become local links. Two-hop fetch via /backend-api/files/{id}/download → signed URL (verified live). all also pulls voice-mode audio (≈162MB of clips in this archive, transcripts already in text — hence the images-only default). Idempotent (skips assets already on disk), atomic writes, 600 perms. Expired/missing assets (old generated images 404) keep their placeholder and are counted as media failed — never fatal.
• Joplin resource upload: the joplin command uploads downloaded media as resources and rewrites media/… links to :/resourceId, so images render inside notes (verified live: resources created with correct mime/size and linked to the note). Resource IDs are tracked per-conversation in the manifest, so re-syncs reuse them instead of duplicating.

Token setup & throughput

• auth --from-browser [brave|chrome|chromium|edge|firefox]: extracts ChatGPT session-token cookies and the Claude sessionKey straight from a local browser's cookie store (via browser-cookie3; defaults to Brave), validates each against the live API, and writes .env. Tokens that fail validation never overwrite a working entry; clear fallback when the browser is absent or the cookie DB is locked.
• Session limiter: --max-conversations N / MAX_CONVERSATIONS_PER_RUN cap downloads per run (per provider); capped-out conversations are reported as "deferred" and resumed next run.
• Polite pacing: REQUEST_DELAY (default 1.0s, ±25% jitter, 0 disables) between consecutive API requests, with the existing 429 backoff as the reactive net.

Re-rendering

• export --force re-exports every conversation even if cached and unchanged, so the whole archive can be re-rendered after a formatting/feature change without cache --clear. Runs as a tracked campaign (a stamped start time in the manifest): each run re-renders the least-recently-exported conversations, the "still to go" count shrinks toward zero, finished providers do no further work, and the campaign auto-completes (reporting "Force re-render complete"). Combines with --max-conversations to spread the work across runs.
• mark_exported now preserves joplin_note_id / joplin_synced_at / joplin_resources across re-exports (refreshing exported_at), so a re-export followed by joplin updates the existing notes instead of creating duplicates.
• Fix: .env is now loaded at the very start of every command, so CACHE_DIR and LOG_FILE are honored (previously the cache silently used ./cache regardless of CACHE_DIR).

Archive hygiene

• prune deletes export files not referenced by the manifest (old-layout trees, no-ID orphans), with listing, confirmation, --dry-run, -y, and empty-directory sweep. Refuses to run when the manifest references no files (so cache --clear + prune can't wipe the archive). First live run removed 420 stale files (9.4 MB).
• doctor verifies manifest ↔ disk integrity: every recorded file_path must exist on disk.

Changed

• ChatGPT role filter that dropped tool/system messages is lifted; all roles route through normal extraction (truly empty messages skip via the empty-content guard).
• BaseProvider.normalize_conversation accepts an optional LossReport parameter.
• ChatGPTProvider.normalize_conversation reads conversation_id as a fallback for id (live detail responses use conversation_id; fixtures use id).
• Config validates EXPORTER_HIDDEN_CONTENT, EXPORTER_DOWNLOAD_MEDIA, MAX_CONVERSATIONS_PER_RUN, and REQUEST_DELAY, and logs the active values at startup.
• New dependency: browser-cookie3==0.20.1.

Fixed

• Custom Instructions (user_editable_context/model_editable_context) were silently dropped from every conversation (parts-vs-direct-fields mismatch).

Migration

• JSON exports: messages contain typed blocks and may omit the legacy content field — external consumers should prefer blocks.
• To re-render existing exports with the current rendering and collapse policy: python -m src.main cache --clear then python -m src.main export, followed by joplin to update notes (and upload media resources). Per-conversation message counts may increase as previously-dropped Custom Instructions, image-only turns, and tool-only turns now appear.

Test suite

• 264 tests, all passing.