AIChatExporter/CHANGELOG.md

Changelog

All notable changes to this project will be documented here. Format follows Keep a Changelog.

[Unreleased]

Fixed

• A single U+0085 in a transcript silently dropped a whole record. Both local providers split session files with str.splitlines(), which breaks not just on \n but on U+0085 (NEL), U+2028 and U+2029 — all of which are legal inside a JSON string and are written literally by Codex (Rust does not escape non-ASCII). One NEL in captured command output shredded one record into unparseable fragments; the parser logged "skipped 3 unparseable line(s)" and lost the record. Found while exporting a real rollout. Both providers now split on \n only, and both have regression tests that write their fixtures with ensure_ascii=False — with json.dumps' default the hazardous characters are escaped and the bug cannot reproduce.
• Deleted uploads are no longer reported as permission errors. ChatGPT's /backend-api/files/{id}/download answers a missing asset with 403 {"detail":"Forbidden"}, which reads like an auth failure and isn't one. Measured live 2026-08-17 across 18 such assets: every one returned 404 {"detail":"File not found"} on /files/{id}, while assets that downloaded fine returned 200 on both in the same session, and ChatGPT-Account-Id made no difference. A 403 is now confirmed against the metadata endpoint before being reported (one extra request on the failure path only, none on success) and a confirmed-missing asset is logged as gone and counted as expired-or-missing. A 403 on an asset that does still exist is left alone as forbidden — that one would be a real problem.
• 4xx errors now report why. _make_request ended non-retryable statuses with raise_for_status(), whose curl_cffi message is HTTP Error {code}: {reason} — and HTTP/2 carries no reason phrase, so a refused request logged as bare HTTP Error 403: and the response body (the only explanation the provider gives) was discarded. The body's detail/error/message is now carried into the ProviderError, redacted and truncated. This is what made the media 403s on GET /backend-api/files/{id}/download undiagnosable.
• redact_secrets missed compound key names. It matched keys exactly, so access_token, api_key, and session-token passed through un-redacted into debug-logged response bodies; matching now applies per word ("keywords", "monkey", "tokenizer" stay intact).
• tests/test_config.py::TestSessionLimiterConfig::test_defaults depended on the developer's .env. load_config() calls load_dotenv(override=False), which re-populated the variable the test had just deleted — so it passed only on a machine with no .env. The test now stubs dotenv discovery.

Added

• Codex CLI provider (--provider codex). Archives local Codex agent transcripts from ~/.codex/sessions/**/rollout-*.jsonl — local-only, like claude-code: no tokens, no rate limits, no ToS exposure. Sessions land in their own top-level AI-Codex Joplin notebook, with the same prose-only default, repo tags (CODEX_REPO_TAG_IGNORE) and multi-root scanning (CODEX_DIR, plus $CODEX_HOME/sessions).

  Codex writes each session twice in one file and the choice between the two layers is the whole design. response_item records are the model-facing wire format, where a tool call arrives as JavaScript (tools.exec_command({...})) because Codex's exec tool is code-mode; event_msg/item_completed records are Codex's own typed items, already decoded into CommandExecution/FileChange/Extension with argv, cwd, exit code and output as fields. Measured over 7 sessions on 0.147.0 (2026-08-18), the typed layer is 1:1 with the raw layer for prose (91 AgentMessage ↔ 91 assistant messages, sharing ids) and additionally omits every piece of harness plumbing — all 51 developer-role messages plus the 7 # AGENTS.md instructions… and 1 <environment_context> injections — which the Claude Code provider has to strip by regex. So the typed layer is parsed for content.

  Its one gap is that it records what ran, not what was attempted: 26 of 180 exec_command calls produced no item (14 sandbox launch failures, 6 user aborts, ~5 still running at turn end, 1 failure). The raw layer is therefore read for a call count only, and placeholders report the shortfall — 3 calls: exec_command ×3 (+2 did not complete) — instead of silently under-reporting. wait calls are process polls, not attempts, and are excluded.

  Reasoning is not exportable from Codex. All 345 reasoning records carry encrypted_content, with summary empty in the raw layer and summary_text/raw_content empty in the typed layer, in every session. It is always dropped and counted; unlike Claude Code, EXPORTER_HIDDEN_CONTENT=full cannot surface it. The sidecar SQLite databases (state_5.sqlite, thread_history_1.sqlite) are deliberately not read: thread_history_projection_state tracks a byte offset into the rollout file, so the JSONL is canonical and the DB derived, and its title column is just the first user message truncated.

  Codex Cloud is out of scope, verified rather than assumed. Cloud tasks are reachable at chatgpt.com/backend-api/api/codex/tasks{,/list} — the same host and /backend-api root the ChatGPT provider already uses — but local CLI sessions are never uploaded there, so it is not an alternative source for these transcripts. codex cloud list confirmed the account holds no cloud tasks. The provider makes no network calls.

• projects command — discover the project IDs your config is missing. CHATGPT_PROJECT_IDS is maintained by hand, and a project missing from it is invisible to the listing pass, so its conversations are never fetched. The command reports every project your conversations belong to, marks which are absent from .env, and prints a paste-ready line (--write applies it). It reads project ids from the conversation listing when they are there and falls back to --deep, one detail request per conversation, when they are not.

• Project attribution now reads the conversation's own gizmo_id. Previously the project name came only from CHATGPT_PROJECT_IDS, so a conversation in a project you had not listed exported into no-project/ even though its payload names its project. The detail response carries gizmo_id, so it is used as a fallback after the listing annotation and the project map — attribution stays correct without maintaining a list, and moving a chat into a new project no longer silently misfiles it. Only g-p- ids count: a custom GPT is not a project and must not become a folder. Each unconfigured project is reported once per run, naming the id to add, because the listing pass still needs CHATGPT_PROJECT_IDS — conversations that live only inside a project never appear in the default listing.

Changed

• Media download failures are bucketed as forbidden (403 — the file record survives) separately from download-error, so the run summary distinguishes it from expired-or-missing (404).

Notes

• ChatGPT media 403s: investigated and closed (2026-08-17). 19 images across 7 conversations would not download. They are unrecoverable, and not because of anything the exporter or the export schedule did. Findings, recorded so this is not re-litigated: uploads do not expire (36/36 sampled images from 2025-09 through 2026-08 are still live, so export cadence is not a factor); the failures split into 7 records that 404 outright and 12 that report state: "ready" with a library_file_id; /files/{id}/download is the endpoint that mints the signed estuary/content URL the web UI fetches, and it refuses the survivors with a bare 403 regardless of headers, Authorization, gizmo_id, conversation_id, Referer, or namespace, while the Library id is rejected as file_not_found. Decisively, those same images render blank in ChatGPT's own UI — nothing is being withheld from the exporter. All the survivors were created 2026-07-14 within minutes of each other yet appear in conversations predating that date, pointing at a Library migration that kept the metadata and lost the bytes.

[0.8.0] - 2026-07-06

Focused on the Claude Code provider after the tool's on-disk layout changed and Claude Code sessions proved hard to find in Joplin.

Added

• Subagent capture. Claude Code now stores subagent (Task-tool) transcripts as separate <session>/subagents/agent-*.jsonl files (with an agent-*.meta.json sidecar). These were previously invisible to the exporter — all delegated work (reviews, research, plans) was lost. Each subagent is now folded into its parent session inline at the Task/Agent call that spawned it, rendered as a collapsible <details> block labeled with its agentType/description. The subagent's own tool traffic is collapsed under the same EXPORTER_HIDDEN_CONTENT policy as the main dialogue; nesting is handled at any depth via toolUseId matching.
• Repo tags in Claude Code titles. Sessions launched from a workspace root all land in one folder-named notebook, so titles now carry the repos each session touched, e.g. Resume StartWRT project work [start-technologies] — visible in the note list and matched by Joplin search. A file's repo is the git repository it lives in (nearest ancestor with a .git), resolved from tool_use paths anywhere in the filesystem — so cross-workspace work is captured and non-repo noise (config dirs, one-off files, reference dirs) is excluded because it isn't a git repo. Frequency-ordered, capped at 3. Escape hatch: CLAUDE_CODE_REPO_TAG_IGNORE (comma-separated repo names). Tags reflect the current git layout, so a since-deleted/moved repo drops from the tag on re-export.
• Multiple projects roots. CLAUDE_CODE_DIR now accepts an os.pathsep-separated list of roots (a single path stays backward compatible), and CLAUDE_CONFIG_DIR's projects/ tree is scanned automatically when set. Sessions from all roots are merged by launch-folder; a session UUID present in two roots keeps the newer-mtime copy.

Changed

• Claude Code gets its own top-level Joplin notebook, AI-ClaudeCode (was nested under AI-Claude alongside Claude web chats, which made dev sessions hard to find). Existing Claude Code notes self-heal into the new notebook on the next joplin run — update_note now sets parent_id, so a changed provider→notebook mapping relocates notes in place instead of duplicating them. After migrating, the emptied AI-Claude/{Myworkspace,Services,…} notebooks can be deleted by hand (Joplin does not auto-remove empty folders).

Notes

• The sibling <session>/tool-results/*.txt sidecars (externalized large tool outputs) are intentionally not captured — tool_result content is collapsed under the default policy anyway.

[0.7.0] - 2026-06-28

Added

• canary command — probes the ChatGPT/Claude web APIs for schema drift against the fields the parser actually depends on (one listing page + one conversation per provider), not the full response shape. Flags the silent-failure risks for a backup tool: a renamed retrieval-tool author bypassing the hidden-content collapse, a new content_type, drifted message fields, or non-empty Claude attachments/files the normalizer ignores. ERROR findings exit non-zero; WARN findings are surfaced but non-fatal. See FUTURE.md §10.
• doctor now reports a real ChatGPT token-health check ("ChatGPT token active") based on the /api/auth/session error field. ChatGPT session tokens are JWEs whose exp is encrypted and unreadable client-side; the previous decode path could never yield an expiry. The honest signal is error == "RefreshAccessTokenError" (verified live), which means the session token is dead even though expires/accessToken still echo stale values. See FUTURE.md §9.

Fixed

• ChatGPTProvider._fetch_access_token now checks the /api/auth/session error field and fails fast with a clear "refresh your token" message. Previously it returned the stale accessToken present on a dead session, producing a confusing downstream 401 instead of an actionable auth error.

Removed

• auth --from-browser and the browser-cookie3 dependency (src/browser_tokens.py). Browser cookie auto-extraction is not viable: modern Chromium App-Bound Encryption (Chrome 127+/current Brave) keys cookies off a SYSTEM-level layer that can't be decrypted off disk without Administrator rights and AV-flagged SYSTEM impersonation, and fails on Brave specifically (symptom: Unable to get key for cookie decryption). Auth is manual (DevTools wizard) only. See FUTURE.md §2 for the full rationale.

[0.6.0] - 2026-06-12

First tagged release. The project was developed through several internal milestones (0.1.0–0.5.0) that were never published; their changes are consolidated here.

Added

Core export & sync

• ChatGPT and Claude export via their internal web APIs, with a local cache/manifest for incremental, resumable sync (only new or updated conversations are fetched; each is written to the manifest immediately).
• Markdown and JSON exporters. Markdown rendering happens at exporter-write time (providers produce typed blocks; exporters render them), which keeps the door open for future Obsidian/HTML output.
• CLI: export, list, cache, doctor, auth, joplin, prune.
• --project filter on export/list/joplin (case-insensitive substring, or none).
• ChatGPT Projects support via CHATGPT_PROJECT_IDS (project conversations are fetched separately from the default listing).

Joplin integration

• joplin command syncs exported Markdown to Joplin as notes; notebooks are created automatically and nested per provider+project. Re-running is safe — the Joplin note ID is stored in the manifest, so notes are updated, not duplicated.
• JOPLIN_API_TOKEN, JOPLIN_API_URL, JOPLIN_REQUEST_TIMEOUT config, with actionable error messages on timeout/connection failure.

Rich content (typed blocks)

• Messages carry an ordered blocks list: text, code, thinking, tool_use, tool_result, citation, image_placeholder, file_placeholder, unknown.
• ChatGPT voice mode: audio_transcription parts render as text; audio asset pointers render as 📎 File attached placeholders with size/duration. Custom Instructions (user_editable_context/model_editable_context) now appear (were silently dropped) with a > ℹ️ Hidden context marker.
• ChatGPT execution_output, system_error, and tether_browsing_display render as tool_result blocks (with tool_name, is_error, optional summary); transient browse spinners skip silently.
• Defensive Claude block extraction (text/thinking/tool_use/tool_result/image, including nested-block flattening).
• _safe_fence picks a backtick fence longer than any run in the content, so embedded triple-backticks can't corrupt rendering (verified live in Joplin).
• Data-loss visibility: a LossReport summary at the end of every export run breaks down unknown blocks and extraction failures by raw type; unknown blocks render as visible > ⚠️ Unsupported content with the raw type and observed keys.

Invisible-content collapse (EXPORTER_HIDDEN_CONTENT=full|placeholder|omit, default placeholder; export --hidden-content)

• Messages that were invisible in the ChatGPT web UI are collapsed to one-line placeholders instead of exported in full. Two triggers: (a) tool-role retrieval dumps identified by author.name (file_search, myfiles_browser) — ChatGPT re-injects the full text of attached/project files on every tool run, measured at 86% of all content bytes across the three largest conversations and not hidden-flagged; (b) messages flagged is_visually_hidden_from_conversation (Custom Instructions). Code-execution results, web-search output, and all dialogue are untouched.
• New collapsed block renders as > 🔧 **Tool output** — file_search (15.0 KB) — omitted (EXPORTER_HIDDEN_CONTENT=full to keep) (ℹ️ Hidden context variant for hidden-flagged messages). The LossReport gains a collapsed by policy section (count per origin + total KB) so the omission stays visible.

Claude Code session provider (--provider claude-code)

• Archives local agent transcripts from ~/.claude/projects/*/*.jsonl (override with CLAUDE_CODE_DIR) — no tokens, no rate limits, no ToS exposure. Prose-only by default per the hidden-content policy: dialogue and deliverables kept; tool traffic grouped into one placeholder per activity run (> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted); thinking dropped (counted in the summary); subagent (isSidechain) and harness (isMeta, snapshots, command tags) records stripped. Titles from the last ai-title record; project from the working-directory basename. Joplin notebooks nest under the AI-Claude parent. Measured: 19MB of JSONL → 804KB of Markdown across 25 sessions.

Binary/image downloads (EXPORTER_DOWNLOAD_MEDIA=images|all|off, default images; export --download-media)

• Downloads ChatGPT conversation assets into a media/ folder beside each export and inlines them — images become real ![](media/…) embeds, files become local links. Two-hop fetch via /backend-api/files/{id}/download → signed URL (verified live). all also pulls voice-mode audio (≈162MB of clips in this archive, transcripts already in text — hence the images-only default). Idempotent (skips assets already on disk), atomic writes, 600 perms. Expired/missing assets (old generated images 404) keep their placeholder and are counted as media failed — never fatal.
• Joplin resource upload: the joplin command uploads downloaded media as resources and rewrites media/… links to :/resourceId, so images render inside notes (verified live: resources created with correct mime/size and linked to the note). Resource IDs are tracked per-conversation in the manifest, so re-syncs reuse them instead of duplicating.

Token setup & throughput

• auth --from-browser [brave|chrome|chromium|edge|firefox]: extracts ChatGPT session-token cookies and the Claude sessionKey straight from a local browser's cookie store (via browser-cookie3; defaults to Brave), validates each against the live API, and writes .env. Tokens that fail validation never overwrite a working entry; clear fallback when the browser is absent or the cookie DB is locked.
• Session limiter: --max-conversations N / MAX_CONVERSATIONS_PER_RUN cap downloads per run (per provider); capped-out conversations are reported as "deferred" and resumed next run.
• Polite pacing: REQUEST_DELAY (default 1.0s, ±25% jitter, 0 disables) between consecutive API requests, with the existing 429 backoff as the reactive net.

Re-rendering

• export --force re-exports every conversation even if cached and unchanged, so the whole archive can be re-rendered after a formatting/feature change without cache --clear. Runs as a tracked campaign (a stamped start time in the manifest): each run re-renders the least-recently-exported conversations, the "still to go" count shrinks toward zero, finished providers do no further work, and the campaign auto-completes (reporting "Force re-render complete"). Combines with --max-conversations to spread the work across runs.
• mark_exported now preserves joplin_note_id / joplin_synced_at / joplin_resources across re-exports (refreshing exported_at), so a re-export followed by joplin updates the existing notes instead of creating duplicates.
• Fix: .env is now loaded at the very start of every command, so CACHE_DIR and LOG_FILE are honored (previously the cache silently used ./cache regardless of CACHE_DIR).

Archive hygiene

• prune deletes export files not referenced by the manifest (old-layout trees, no-ID orphans), with listing, confirmation, --dry-run, -y, and empty-directory sweep. Refuses to run when the manifest references no files (so cache --clear + prune can't wipe the archive). First live run removed 420 stale files (9.4 MB).
• doctor verifies manifest ↔ disk integrity: every recorded file_path must exist on disk.

Changed

• ChatGPT role filter that dropped tool/system messages is lifted; all roles route through normal extraction (truly empty messages skip via the empty-content guard).
• BaseProvider.normalize_conversation accepts an optional LossReport parameter.
• ChatGPTProvider.normalize_conversation reads conversation_id as a fallback for id (live detail responses use conversation_id; fixtures use id).
• Config validates EXPORTER_HIDDEN_CONTENT, EXPORTER_DOWNLOAD_MEDIA, MAX_CONVERSATIONS_PER_RUN, and REQUEST_DELAY, and logs the active values at startup.
• New dependency: browser-cookie3==0.20.1.

Fixed

• Custom Instructions (user_editable_context/model_editable_context) were silently dropped from every conversation (parts-vs-direct-fields mismatch).

Migration

• JSON exports: messages contain typed blocks and may omit the legacy content field — external consumers should prefer blocks.
• To re-render existing exports with the current rendering and collapse policy: python -m src.main cache --clear then python -m src.main export, followed by joplin to update notes (and upload media resources). Per-conversation message counts may increase as previously-dropped Custom Instructions, image-only turns, and tool-only turns now appear.

Test suite

• 264 tests, all passing.