Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 JesseMarkowitz
|
9e1a8ab7cb | ||
|
JesseMarkowitzandClaude Sonnet 4.6
|
557994f7d9 | ||
|
JesseMarkowitzandClaude Sonnet 4.6
|
e9b2e42893 | ||
|
JesseMarkowitzandClaude Opus 4.7
|
68e8d532be | ||
|
JesseMarkowitzandClaude Opus 4.7
|
473d02f71a | ||
 Jesse.MarkowitzandClaude Sonnet 4.6
|
4798edcea7 |
@@ -36,6 +36,31 @@ EXPORT_DIR=./exports
|
||||
# provider/year → exports/claude/2024/file.md (ignores projects)
|
||||
OUTPUT_STRUCTURE=provider/project/year
|
||||
|
||||
# What to do with content that was invisible in the provider's web UI
|
||||
# (file-retrieval tool dumps, hidden context like Custom Instructions).
|
||||
# These dumps can be 90% of a conversation's bytes. Options:
|
||||
# placeholder (default) → one-line placeholder with tool name and size
|
||||
# full → keep everything (pre-v0.6.0 behavior)
|
||||
# omit → drop entirely (still counted in the run summary)
|
||||
EXPORTER_HIDDEN_CONTENT=placeholder
|
||||
|
||||
# Download conversation assets (images, audio) next to the Markdown, into a
|
||||
# media/ folder, and inline them. Options:
|
||||
# images (default) → images only
|
||||
# all → also audio/voice clips and other files
|
||||
# off → keep text placeholders, download nothing
|
||||
# Downloaded media is uploaded to Joplin as resources on the next `joplin` run.
|
||||
EXPORTER_DOWNLOAD_MEDIA=images
|
||||
|
||||
# Cap how many conversations are downloaded per export run (per provider).
|
||||
# Runs are resumable — a capped run continues where it stopped next time.
|
||||
# Keeps big backfills from looking like scraper traffic. Unset = unlimited.
|
||||
#MAX_CONVERSATIONS_PER_RUN=25
|
||||
|
||||
# Seconds between consecutive API requests (small random jitter is added).
|
||||
# Default 1.0; set 0 to disable pacing.
|
||||
#REQUEST_DELAY=1.0
|
||||
|
||||
# --- Joplin ---
|
||||
# Automate importing exported conversations into Joplin as notes.
|
||||
# Requires Joplin desktop running with the Web Clipper service enabled.
|
||||
|
||||
+61
-14
@@ -3,19 +3,66 @@
|
||||
All notable changes to this project will be documented here.
|
||||
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
|
||||
## [0.2.0] - Unreleased
|
||||
### Added
|
||||
- Joplin import automation: `joplin` command syncs exported Markdown files to Joplin as notes
|
||||
- Notebooks created automatically per provider+project (`ChatGPT - My Project`, etc.)
|
||||
- Re-running is safe: notes are updated, not duplicated (Joplin note ID stored in manifest)
|
||||
- `JOPLIN_API_TOKEN`, `JOPLIN_API_URL`, `JOPLIN_REQUEST_TIMEOUT` config variables
|
||||
- Configurable request timeout with clear error messages and actionable hints on timeout
|
||||
- `--project` filter on `export` and `list` commands (case-insensitive substring or `none`)
|
||||
- ChatGPT Projects support via `CHATGPT_PROJECT_IDS` env var
|
||||
## [0.6.0] - 2026-06-12
|
||||
|
||||
First tagged release. The project was developed through several internal
|
||||
milestones (0.1.0–0.5.0) that were never published; their changes are
|
||||
consolidated here.
|
||||
|
||||
## [0.1.0] - Unreleased
|
||||
### Added
|
||||
- Initial implementation: ChatGPT and Claude export via internal web APIs
|
||||
- Markdown and JSON exporters
|
||||
- Local cache/manifest for incremental sync
|
||||
- CLI with export, list, cache, doctor, and auth commands
|
||||
|
||||
**Core export & sync**
|
||||
- ChatGPT and Claude export via their internal web APIs, with a local cache/manifest for incremental, resumable sync (only new or updated conversations are fetched; each is written to the manifest immediately).
|
||||
- Markdown and JSON exporters. Markdown rendering happens at exporter-write time (providers produce typed blocks; exporters render them), which keeps the door open for future Obsidian/HTML output.
|
||||
- CLI: `export`, `list`, `cache`, `doctor`, `auth`, `joplin`, `prune`.
|
||||
- `--project` filter on `export`/`list`/`joplin` (case-insensitive substring, or `none`).
|
||||
- ChatGPT Projects support via `CHATGPT_PROJECT_IDS` (project conversations are fetched separately from the default listing).
|
||||
|
||||
**Joplin integration**
|
||||
- `joplin` command syncs exported Markdown to Joplin as notes; notebooks are created automatically and nested per provider+project. Re-running is safe — the Joplin note ID is stored in the manifest, so notes are updated, not duplicated.
|
||||
- `JOPLIN_API_TOKEN`, `JOPLIN_API_URL`, `JOPLIN_REQUEST_TIMEOUT` config, with actionable error messages on timeout/connection failure.
|
||||
|
||||
**Rich content (typed blocks)**
|
||||
- Messages carry an ordered `blocks` list: text, code, thinking, tool_use, tool_result, citation, image_placeholder, file_placeholder, unknown.
|
||||
- ChatGPT voice mode: `audio_transcription` parts render as text; audio asset pointers render as `📎 File attached` placeholders with size/duration. Custom Instructions (`user_editable_context`/`model_editable_context`) now appear (were silently dropped) with a `> ℹ️ Hidden context` marker.
|
||||
- ChatGPT `execution_output`, `system_error`, and `tether_browsing_display` render as `tool_result` blocks (with `tool_name`, `is_error`, optional `summary`); transient browse spinners skip silently.
|
||||
- Defensive Claude block extraction (text/thinking/tool_use/tool_result/image, including nested-block flattening).
|
||||
- `_safe_fence` picks a backtick fence longer than any run in the content, so embedded triple-backticks can't corrupt rendering (verified live in Joplin).
|
||||
- Data-loss visibility: a `LossReport` summary at the end of every `export` run breaks down `unknown blocks` and `extraction failures` by raw type; `unknown` blocks render as visible `> ⚠️ Unsupported content` with the raw type and observed keys.
|
||||
|
||||
**Invisible-content collapse (`EXPORTER_HIDDEN_CONTENT=full|placeholder|omit`, default `placeholder`; `export --hidden-content`)**
|
||||
- Messages that were invisible in the ChatGPT web UI are collapsed to one-line placeholders instead of exported in full. Two triggers: (a) tool-role retrieval dumps identified by `author.name` (`file_search`, `myfiles_browser`) — ChatGPT re-injects the full text of attached/project files on every tool run, measured at 86% of all content bytes across the three largest conversations and **not** hidden-flagged; (b) messages flagged `is_visually_hidden_from_conversation` (Custom Instructions). Code-execution results, web-search output, and all dialogue are untouched.
|
||||
- New `collapsed` block renders as `> 🔧 **Tool output** — file_search (15.0 KB) — omitted (EXPORTER_HIDDEN_CONTENT=full to keep)` (ℹ️ Hidden context variant for hidden-flagged messages). The `LossReport` gains a `collapsed by policy` section (count per origin + total KB) so the omission stays visible.
|
||||
|
||||
**Claude Code session provider (`--provider claude-code`)**
|
||||
- Archives local agent transcripts from `~/.claude/projects/*/*.jsonl` (override with `CLAUDE_CODE_DIR`) — no tokens, no rate limits, no ToS exposure. Prose-only by default per the hidden-content policy: dialogue and deliverables kept; tool traffic grouped into one placeholder per activity run (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`); thinking dropped (counted in the summary); subagent (`isSidechain`) and harness (`isMeta`, snapshots, command tags) records stripped. Titles from the last `ai-title` record; project from the working-directory basename. Joplin notebooks nest under the `AI-Claude` parent. Measured: 19MB of JSONL → 804KB of Markdown across 25 sessions.
|
||||
|
||||
**Binary/image downloads (`EXPORTER_DOWNLOAD_MEDIA=images|all|off`, default `images`; `export --download-media`)**
|
||||
- Downloads ChatGPT conversation assets into a `media/` folder beside each export and inlines them — images become real `` embeds, files become local links. Two-hop fetch via `/backend-api/files/{id}/download` → signed URL (verified live). `all` also pulls voice-mode audio (≈162MB of clips in this archive, transcripts already in text — hence the images-only default). Idempotent (skips assets already on disk), atomic writes, `600` perms. Expired/missing assets (old generated images 404) keep their placeholder and are counted as `media failed` — never fatal.
|
||||
- Joplin resource upload: the `joplin` command uploads downloaded media as resources and rewrites `media/…` links to `:/resourceId`, so images render inside notes (verified live: resources created with correct mime/size and linked to the note). Resource IDs are tracked per-conversation in the manifest, so re-syncs reuse them instead of duplicating.
|
||||
|
||||
**Token setup & throughput**
|
||||
- `auth --from-browser [brave|chrome|chromium|edge|firefox]`: extracts ChatGPT session-token cookies and the Claude `sessionKey` straight from a local browser's cookie store (via `browser-cookie3`; defaults to Brave), validates each against the live API, and writes `.env`. Tokens that fail validation never overwrite a working entry; clear fallback when the browser is absent or the cookie DB is locked.
|
||||
- Session limiter: `--max-conversations N` / `MAX_CONVERSATIONS_PER_RUN` cap downloads per run (per provider); capped-out conversations are reported as "deferred" and resumed next run.
|
||||
- Polite pacing: `REQUEST_DELAY` (default 1.0s, ±25% jitter, `0` disables) between consecutive API requests, with the existing 429 backoff as the reactive net.
|
||||
|
||||
**Archive hygiene**
|
||||
- `prune` deletes export files not referenced by the manifest (old-layout trees, no-ID orphans), with listing, confirmation, `--dry-run`, `-y`, and empty-directory sweep. Refuses to run when the manifest references no files (so `cache --clear` + `prune` can't wipe the archive). First live run removed 420 stale files (9.4 MB).
|
||||
- `doctor` verifies manifest ↔ disk integrity: every recorded `file_path` must exist on disk.
|
||||
|
||||
### Changed
|
||||
- ChatGPT role filter that dropped `tool`/`system` messages is **lifted**; all roles route through normal extraction (truly empty messages skip via the empty-content guard).
|
||||
- `BaseProvider.normalize_conversation` accepts an optional `LossReport` parameter.
|
||||
- `ChatGPTProvider.normalize_conversation` reads `conversation_id` as a fallback for `id` (live detail responses use `conversation_id`; fixtures use `id`).
|
||||
- `Config` validates `EXPORTER_HIDDEN_CONTENT`, `EXPORTER_DOWNLOAD_MEDIA`, `MAX_CONVERSATIONS_PER_RUN`, and `REQUEST_DELAY`, and logs the active values at startup.
|
||||
- New dependency: `browser-cookie3==0.20.1`.
|
||||
|
||||
### Fixed
|
||||
- Custom Instructions (`user_editable_context`/`model_editable_context`) were silently dropped from every conversation (parts-vs-direct-fields mismatch).
|
||||
|
||||
### Migration
|
||||
- JSON exports: messages contain typed `blocks` and may omit the legacy `content` field — external consumers should prefer `blocks`.
|
||||
- To re-render existing exports with the current rendering and collapse policy: `python -m src.main cache --clear` then `python -m src.main export`, followed by `joplin` to update notes (and upload media resources). Per-conversation message counts may increase as previously-dropped Custom Instructions, image-only turns, and tool-only turns now appear.
|
||||
|
||||
### Test suite
|
||||
- 264 tests, all passing.
|
||||
|
||||
@@ -7,81 +7,261 @@ of these additions straightforward.
|
||||
**Completed:**
|
||||
- v0.1.0 — Core export: ChatGPT + Claude, incremental sync, Markdown + JSON output
|
||||
- v0.2.0 — Joplin import automation (`joplin` command, create/update notes, notebook auto-creation)
|
||||
- v0.4.0 — Rich content support: typed message blocks (text, code, thinking, tool_use, tool_result, image_placeholder, file_placeholder, unknown); ChatGPT voice transcripts as text + audio placeholders; Custom Instructions extraction; data-loss visibility via `LossReport` summary and visible `unknown` blocks
|
||||
- v0.5.0 — Nested Joplin notebooks, date-prefixed note titles, flat year folders
|
||||
- v0.6.0 — Collapse tool retrieval dumps & hidden context (`EXPORTER_HIDDEN_CONTENT` policy; roadmap item 1); session limiter + request pacing (`MAX_CONVERSATIONS_PER_RUN`, `REQUEST_DELAY`; roadmap item 6)
|
||||
|
||||
---
|
||||
|
||||
## Export `--force` Flag (v0.2.x)
|
||||
# Roadmap (decided 2026-06-12)
|
||||
|
||||
Add `--force` to the `export` command to re-export already-cached conversations
|
||||
without permanently clearing the entire manifest. Useful for re-generating files
|
||||
after changing the Markdown template or output structure.
|
||||
Priorities reflect the tool's primary purpose — a trustworthy backup so that
|
||||
conversation data is not lost if a provider account is ever closed — plus the
|
||||
day-to-day friction of weekly ChatGPT token refresh, and the long-term goal
|
||||
of running headless as a StartOS service.
|
||||
|
||||
Implementation: pass a `force=True` flag to `cache.get_new_or_updated()`, which
|
||||
returns all conversations regardless of cache state when force is True.
|
||||
**Now (in order):**
|
||||
1. ~~Collapse tool retrieval dumps & hidden context~~ — **shipped in v0.6.0**
|
||||
(full-archive re-export still pending; see entry below)
|
||||
2. ~~Brave cookie auto-extraction~~ — **shipped in v0.6.0** (note: runs on
|
||||
whichever machine hosts the browser — Brave is not on this Linux box)
|
||||
3. ~~Claude Code session provider~~ — **shipped in v0.6.0**
|
||||
4. ~~Archive hygiene — `prune` + `doctor` integrity check~~ — **shipped in v0.6.0**
|
||||
|
||||
Current workaround: `python -m src.main cache --clear` then re-run export.
|
||||
**Soon, but later:**
|
||||
|
||||
## Joplin `--force` Flag (v0.2.x)
|
||||
5. ~~Binary content downloads~~ — **shipped in v0.6.0**
|
||||
6. ~~Per-session download limiter + polite pacing~~ — **shipped in v0.6.0**
|
||||
7. Scheduled / watch mode → StartOS service packaging (long-term destination)
|
||||
|
||||
Similarly, add `--force` to the `joplin` command to re-sync all cached
|
||||
conversations to Joplin regardless of whether they've been synced before.
|
||||
Useful after making formatting changes to the Markdown exporter.
|
||||
|
||||
Implementation: in `get_joplin_pending()`, return all entries that have a
|
||||
`file_path` when `force=True`, ignoring `joplin_synced_at`.
|
||||
|
||||
## Per-Conversation Cache Reset (v0.2.x)
|
||||
|
||||
Add `cache --reset --conversation <id>` to force re-export or re-sync of a
|
||||
single conversation without clearing the entire provider cache.
|
||||
|
||||
Current workaround: manually edit `~/.ai-chat-exporter/manifest.json` and
|
||||
delete the entry, then re-run export.
|
||||
**Deprioritized** (entries kept at the bottom of this file; revisit on
|
||||
demand): `--force` flags, per-conversation cache reset, official export-ZIP
|
||||
fallback, o1/o3 reasoning reclassification, Obsidian output, token expiry
|
||||
notifications (largely superseded by cookie auto-extraction), search command.
|
||||
Additional web providers (Gemini/Grok/Perplexity) are explicitly out of
|
||||
scope — no significant usage to archive.
|
||||
|
||||
---
|
||||
|
||||
## Official API Fallback (v0.3.0)
|
||||
## 1. Collapse Tool Retrieval Dumps & Hidden Context — SHIPPED v0.6.0
|
||||
|
||||
If the unofficial internal web API approach breaks, migrate to official export
|
||||
file parsing as a fallback:
|
||||
- ChatGPT: parse `conversations.json` from Settings → Export Data
|
||||
- Claude: parse `conversations.json` from Settings → Privacy → Export Data
|
||||
**Implemented 2026-06-12** as designed below, with one scoping correction
|
||||
from live recon: the retrieval dumps are NOT flagged
|
||||
`is_visually_hidden_from_conversation` — `author.name == "file_search"` is
|
||||
the discriminator (the hidden flag only marks Custom Instructions and small
|
||||
system stubs). Verified live: worst files shrink 93% (524KB → 36KB).
|
||||
Remaining step: full-archive re-export (`cache --clear` + `export`) +
|
||||
Joplin re-sync, at the user's chosen time.
|
||||
|
||||
The `BaseProvider` abstract class is intentionally designed so that a
|
||||
`FileProvider` subclass can implement the same interface
|
||||
(`list_conversations`, `get_conversation`, `normalize_conversation`)
|
||||
without any changes to cache, exporters, or CLI code.
|
||||
**Problem (measured 2026-06-12 against a full fresh export):** 45% of the
|
||||
entire 11.2MB archive (260 files) is tool-role messages; 29 files are
|
||||
majority tool-dump. Worst case: a 524KB conversation where 495KB (94%, 64
|
||||
messages) is ChatGPT's file-retrieval tool re-injecting the full text of
|
||||
the user's own attached documents, the same files dumped dozens of times
|
||||
per conversation. These messages were invisible in the ChatGPT web UI;
|
||||
they appear in exports because v0.4.0 lifted the role filter to fix silent
|
||||
data loss. Custom Instructions hidden-context blocks are a minor secondary
|
||||
case (~2KB, once per conversation) — the originally planned
|
||||
`EXPORTER_INCLUDE_HIDDEN_CONTEXT` toggle alone would not help: the worst
|
||||
file contains zero hidden-context blocks.
|
||||
|
||||
To add this: implement `src/providers/file_chatgpt.py` and
|
||||
`src/providers/file_claude.py`, then add `--input-file` flag to the
|
||||
export command to accept a pre-downloaded export ZIP or JSON.
|
||||
**Fix: collapse, don't drop** (consistent with the no-silent-drop rule):
|
||||
|
||||
---
|
||||
- `EXPORTER_HIDDEN_CONTENT=full|placeholder|omit` env var, default
|
||||
`placeholder`, plus a `--hidden-content` CLI override on `export`.
|
||||
- `placeholder` renders affected messages as one line with type and size:
|
||||
`> 🔧 Tool output (file_search, 24KB) — omitted
|
||||
(EXPORTER_HIDDEN_CONTENT=full to keep)`. Expected effect: archive
|
||||
roughly halves; worst files shrink ~90%.
|
||||
- **Scope — collapse:** (a) tool-role retrieval dumps, identified by raw
|
||||
`author.name` (`file_search`, `myfiles_browser`, …) in the API response
|
||||
(the rendered Markdown only shows a generic "🔧 Tool" label, so the
|
||||
decision must happen in the provider, not the renderer); (b) messages
|
||||
flagged `is_visually_hidden_from_conversation`, including
|
||||
`user_editable_context` / `model_editable_context` (Custom
|
||||
Instructions) — subsumes the old suppress-hidden-context idea.
|
||||
- **Scope — keep at full size:** code-execution `tool_result` blocks and
|
||||
web-search results; those are usually content the user wants.
|
||||
- Count collapsed messages in the post-export summary so the omission
|
||||
stays visible (mirror the LossReport presentation, but as intentional
|
||||
policy, not loss).
|
||||
|
||||
## Rich Content Support (v0.4.0)
|
||||
Re-export workflow after shipping: `cache --clear` + `export` (same as
|
||||
the v0.4.0 migration).
|
||||
|
||||
Currently only text content is exported. Future versions should handle:
|
||||
## 2. Brave Cookie Auto-Extraction — SHIPPED v0.6.0
|
||||
|
||||
### Claude
|
||||
- Artifacts (code, documents, HTML) — export as separate files, link from Markdown
|
||||
- Uploaded images — download and embed or link
|
||||
- Extended thinking/reasoning blocks — include as collapsible sections
|
||||
- Tool call results and web search citations — include as footnotes or appendices
|
||||
**Implemented 2026-06-12**: `auth --from-browser [browser]`
|
||||
(`src/browser_tokens.py`, browser-cookie3 dependency, defaults to brave;
|
||||
chrome/chromium/edge/firefox also supported). Extracted tokens are validated
|
||||
against the live API before `.env` is touched. **Discovered during
|
||||
implementation: Brave is not installed on this machine** — the logged-in
|
||||
browser lives elsewhere, so the flag only helps when the exporter runs on
|
||||
that machine (verified the graceful-failure path here; the happy path is
|
||||
covered by mocked tests). This strengthens the case for the StartOS
|
||||
token-push mechanism (entry 8).
|
||||
|
||||
### ChatGPT
|
||||
- DALL-E generated images — download and embed or link
|
||||
- Code Interpreter outputs — export code and results
|
||||
- File attachments — download and reference
|
||||
- Voice transcripts — include as text
|
||||
Pull session tokens directly from the Brave browser profile instead of the
|
||||
weekly manual DevTools dance. `auth` gains an "extract from browser" path
|
||||
(with the manual flow kept as fallback); a later iteration could let
|
||||
`doctor` or a 401 handler suggest/perform a re-extract automatically.
|
||||
|
||||
Implementation note: the normalized message schema already includes a
|
||||
`content_type` field placeholder. When this work begins, extend the schema
|
||||
rather than replacing it. Non-text content already logs a WARNING when
|
||||
encountered so users can see what was skipped.
|
||||
- Brave on Linux follows the Chromium pattern: cookies SQLite at
|
||||
`~/.config/BraveSoftware/Brave-Browser/Default/Cookies`, values
|
||||
AES-128-CBC encrypted with a key derived from the OS keyring secret
|
||||
("Brave Safe Storage" via SecretService/kwallet, or the `peanuts` v10
|
||||
fallback). Well-trodden territory — evaluate a small dependency
|
||||
(`browser_cookie3`-style) vs. a focused in-repo implementation.
|
||||
- Extract `__Secure-next-auth.session-token.0` / `.1` from `chatgpt.com`
|
||||
and `sessionKey` from `claude.ai`; write to `.env` through the existing
|
||||
auth-wizard write path (tokens never echoed).
|
||||
- Brave must be the user's logged-in browser (it is); detect a locked /
|
||||
unreadable cookie DB (Brave running with the DB exclusively locked) and
|
||||
fall back to the manual flow with a clear message.
|
||||
- Does **not** solve headless/StartOS — no browser on the server. That
|
||||
needs a token-push mechanism, tracked under the StartOS entry.
|
||||
|
||||
---
|
||||
## 3. Claude Code Session Provider — SHIPPED v0.6.0
|
||||
|
||||
## Scheduled / Watch Mode (v0.5.0)
|
||||
**Implemented 2026-06-12** as designed below (`src/providers/claude_code.py`,
|
||||
`--provider claude-code`, `CLAUDE_CODE_DIR` override). Additional findings
|
||||
during implementation: `isSidechain` records are subagent transcripts (skipped),
|
||||
`isMeta` marks harness-generated user records (skipped), and listing/normalized
|
||||
`updated_at` must both use file mtime or the cache would re-export every
|
||||
session every run. First export: 25 sessions, 19MB JSONL → 804KB Markdown.
|
||||
|
||||
Archive local Claude Code session transcripts. No tokens, no rate limits,
|
||||
no ToS risk — the data is already on disk but lives in a single JSONL per
|
||||
session that Claude Code may clean up, and it contains deliverables
|
||||
(reviews, plans, analyses) that exist nowhere else.
|
||||
|
||||
Decisions (2026-06-12):
|
||||
- **Rendering: prose-only.** Keep user prompts and assistant text
|
||||
(including full deliverable write-ups); collapse tool activity to
|
||||
one-line placeholders (`> 🔧 Tool activity — 14 calls (Read ×9, Bash ×2),
|
||||
86KB — omitted`); **exclude thinking blocks**.
|
||||
- **Joplin: sync enabled.** Each coding project becomes a notebook nested
|
||||
under an **"AI-Claude"** parent notebook (nested-notebook support shipped
|
||||
in v0.5.0).
|
||||
|
||||
Data facts (measured 2026-06-12):
|
||||
- Source: `~/.claude/projects/<munged-cwd>/<session-uuid>.jsonl`.
|
||||
Currently 29 sessions, 18.8MB total, largest 3.8MB.
|
||||
- Representative 2.1MB session: tool_result 436KB, tool_use 122KB,
|
||||
thinking 104KB, dialogue prose only ~24KB (~4%) — collapsing tool
|
||||
activity is what makes these exports readable.
|
||||
- Record types: `user` / `assistant` (Anthropic-style `message.content`
|
||||
block arrays) plus harness records: `ai-title` (use for note title and
|
||||
filename slug), `last-prompt`, `file-history-snapshot`, `attachment`,
|
||||
`permission-mode`, `system` (skip). Strip harness noise from user
|
||||
messages (`<local-command-caveat>`, `<command-name>` blocks).
|
||||
|
||||
Implementation shape: new `src/providers/claude_code.py` implementing the
|
||||
`BaseProvider` interface — `list_conversations` scans project dirs,
|
||||
`get_conversation` parses the JSONL, `normalize_conversation` maps onto the
|
||||
existing block schema (content is already block-shaped: text / tool_use /
|
||||
tool_result / thinking). Incremental sync via file mtime/size recorded in
|
||||
the existing manifest. Project name derives from the munged cwd dirname.
|
||||
|
||||
## 4. Archive Hygiene: `prune` Command + Manifest Integrity — SHIPPED v0.6.0
|
||||
|
||||
**Implemented 2026-06-12** as designed below, plus an empty-manifest guard
|
||||
(refuses to prune right after `cache --clear`). First live run removed 420
|
||||
stale files (9.4 MB, old-layout trees + `_.md` orphans); doctor now reports
|
||||
manifest↔disk integrity (293/293 after the run).
|
||||
|
||||
A backup is only trustworthy if the on-disk tree matches the manifest.
|
||||
Observed 2026-06-12: pre-v0.5.0 layout trees (`tspc-expertcouncil/2025/`)
|
||||
and `_.md` no-ID orphans (from the empty-conversation-id bug fixed in
|
||||
v0.4.1) sit alongside current exports and would double-sync into Joplin.
|
||||
|
||||
- `prune` command: delete export files not referenced by the manifest.
|
||||
`--dry-run` (default off, but always print the list before deleting)
|
||||
shows what would be removed and why (old layout / orphan / unknown).
|
||||
- `doctor` extension: verify every manifest entry's `file_path` exists on
|
||||
disk; report missing files (re-export candidates) and unreferenced files
|
||||
(prune candidates).
|
||||
|
||||
## 5. Binary Content Downloads — SHIPPED v0.6.0
|
||||
|
||||
**Implemented 2026-06-12** (`src/media.py`, `EXPORTER_DOWNLOAD_MEDIA`,
|
||||
`download_asset`/`parse_asset_file_id` on the ChatGPT provider, Joplin
|
||||
`create_resource` + `upload_media_and_rewrite`). Live recon settled the
|
||||
download mechanism: `GET /backend-api/files/{id}/download` returns a signed
|
||||
`download_url`; a second GET yields the bytes (works for user uploads;
|
||||
older AI-generated images 404 — expired server-side, handled gracefully).
|
||||
Asset refs come in three shapes — `sediment://file_…`,
|
||||
`sediment://<hash>#file_…#p_N.png` (generated), `file-service://…`. Archive
|
||||
scan: 14 images (4 uploads / 10 generated) + 556 audio clips ≈162MB, so
|
||||
images-only is the default and audio is opt-in via `all`.
|
||||
|
||||
Original notes below.
|
||||
|
||||
**Priority note (2026-06-12): "later, but soon" — under the
|
||||
backup-if-account-closes goal, embedded images are part of the data that
|
||||
would be lost; placeholders alone don't preserve them.**
|
||||
|
||||
v0.4.0 ships placeholders for images and audio assets but does not download
|
||||
the binary content. The `_safe_fence`-wrapped placeholders include the asset
|
||||
reference (`sediment://...` or `file-service://...`), MIME type, size, and
|
||||
duration where available; the actual bytes are not preserved.
|
||||
|
||||
Next steps:
|
||||
- Download attached images alongside the Markdown export, save under a
|
||||
`media/` sibling directory with a stable filename derived from the asset
|
||||
reference.
|
||||
- Replace `image_placeholder` rendering with an inline ``
|
||||
reference once the file is on disk.
|
||||
- Joplin integration: upload binaries as Joplin resources via `POST /resources`,
|
||||
rewrite the rendered Markdown to use `:/resourceId` references, and track
|
||||
the resource ID in the cache manifest so re-syncs stay idempotent.
|
||||
- DALL-E images on the assistant side: not observed in this user's data; the
|
||||
code path exists (`source = "model_generated"`) but is untested.
|
||||
|
||||
The block-level schema is already in place — only the file-fetch + rewrite
|
||||
layer needs to be added. See the `image_placeholder` and `file_placeholder`
|
||||
block definitions in `src/blocks.py`.
|
||||
|
||||
## 6. Per-Session Download Limiter — SHIPPED v0.6.0
|
||||
|
||||
**Implemented 2026-06-12** as designed below: `--max-conversations N` /
|
||||
`MAX_CONVERSATIONS_PER_RUN` session cap with deferred-count reporting, and
|
||||
`REQUEST_DELAY` pacing (default 1.0s ±25% jitter) in `BaseProvider._request`.
|
||||
Verified live: a 2-pending run with cap 1 exported one, deferred one, and
|
||||
the re-run picked it up.
|
||||
|
||||
Cap how many conversations are downloaded in a single `export` run so the
|
||||
tool never hammers the ChatGPT/Claude internal APIs with a large burst —
|
||||
most importantly on the very first export, which otherwise fetches the
|
||||
entire conversation history in one session. Because every run is resumable
|
||||
(the manifest records each conversation immediately), a capped run simply
|
||||
exports the first N pending conversations and the next run picks up where
|
||||
it left off. This keeps traffic looking like a human-paced session rather
|
||||
than a scraper, reducing the risk of rate limiting or account flags.
|
||||
|
||||
Two complementary pieces:
|
||||
|
||||
1. **Session cap** — `--max-conversations N` flag (and
|
||||
`MAX_CONVERSATIONS_PER_RUN` env default). Implementation: in the
|
||||
`export` command, slice the pending list after the cache filter:
|
||||
`to_export = to_export[:n]`. On exit, print exported-vs-remaining
|
||||
counts (reuse the message format from the 429 early-exit path) and
|
||||
remind the user to re-run to continue.
|
||||
2. **Polite pacing** — `REQUEST_DELAY` env var (seconds, with small
|
||||
random jitter) slept between per-conversation detail fetches in
|
||||
`BaseProvider`, so even a capped run doesn't fire requests
|
||||
back-to-back. The existing 429 backoff in `_request` stays as the
|
||||
reactive safety net.
|
||||
|
||||
Note: the conversation *listing* (paginated, 100/page) still runs in full
|
||||
each time so the cache comparison works — the cap applies to the heavy
|
||||
per-conversation detail fetches, which dominate request volume.
|
||||
|
||||
Together with watch mode, this is a stepping stone to the StartOS service:
|
||||
a scheduled, capped, politely-paced export is the traffic profile a
|
||||
headless deployment needs.
|
||||
|
||||
## 7. Scheduled / Watch Mode
|
||||
|
||||
Add a `watch` command (or cron integration helper) to run exports automatically
|
||||
on a schedule:
|
||||
@@ -98,9 +278,84 @@ Implementation: simple loop with `time.sleep()`, or emit a crontab entry
|
||||
string that calls the export and joplin commands in sequence. A `--once`
|
||||
flag would do a single run then exit (useful for cron itself).
|
||||
|
||||
Stepping stone to the StartOS service (below).
|
||||
|
||||
## 8. StartOS Service Packaging (long-term destination)
|
||||
|
||||
Run the exporter headless as a StartOS service: scheduled export + sync to
|
||||
a Joplin Server instance, so the backup happens without a desktop in the
|
||||
loop.
|
||||
|
||||
- Building blocks needed first: watch mode (#7), session limiter (#6),
|
||||
and a Joplin Server (vs. desktop Web Clipper) sync target.
|
||||
- **Hard problem, flagged early:** session-token freshness without a
|
||||
browser. ChatGPT tokens last ~7 days; there is no browser on the server
|
||||
to re-extract from. Candidate mechanisms: a service config action the
|
||||
user pastes fresh tokens into periodically, or a future browser
|
||||
extension that pushes tokens to the server. Cookie auto-extraction (#2)
|
||||
only solves the desktop case.
|
||||
|
||||
---
|
||||
|
||||
## Obsidian Vault Output (v0.5.0)
|
||||
# Deprioritized
|
||||
|
||||
Kept for reference; not on the active roadmap.
|
||||
|
||||
## Export `--force` Flag
|
||||
|
||||
Add `--force` to the `export` command to re-export already-cached conversations
|
||||
without permanently clearing the entire manifest. Useful for re-generating files
|
||||
after changing the Markdown template or output structure.
|
||||
|
||||
Implementation: pass a `force=True` flag to `cache.get_new_or_updated()`, which
|
||||
returns all conversations regardless of cache state when force is True.
|
||||
|
||||
Current workaround: `python -m src.main cache --clear` then re-run export.
|
||||
|
||||
## Joplin `--force` Flag
|
||||
|
||||
Similarly, add `--force` to the `joplin` command to re-sync all cached
|
||||
conversations to Joplin regardless of whether they've been synced before.
|
||||
Useful after making formatting changes to the Markdown exporter.
|
||||
|
||||
Implementation: in `get_joplin_pending()`, return all entries that have a
|
||||
`file_path` when `force=True`, ignoring `joplin_synced_at`.
|
||||
|
||||
## Per-Conversation Cache Reset
|
||||
|
||||
Add `cache --reset --conversation <id>` to force re-export or re-sync of a
|
||||
single conversation without clearing the entire provider cache.
|
||||
|
||||
Current workaround: manually edit `~/.ai-chat-exporter/manifest.json` and
|
||||
delete the entry, then re-run export.
|
||||
|
||||
## Official API Fallback
|
||||
|
||||
If the unofficial internal web API approach breaks, migrate to official export
|
||||
file parsing as a fallback:
|
||||
- ChatGPT: parse `conversations.json` from Settings → Export Data
|
||||
- Claude: parse `conversations.json` from Settings → Privacy → Export Data
|
||||
|
||||
The `BaseProvider` abstract class is intentionally designed so that a
|
||||
`FileProvider` subclass can implement the same interface
|
||||
(`list_conversations`, `get_conversation`, `normalize_conversation`)
|
||||
without any changes to cache, exporters, or CLI code.
|
||||
|
||||
To add this: implement `src/providers/file_chatgpt.py` and
|
||||
`src/providers/file_claude.py`, then add `--input-file` flag to the
|
||||
export command to accept a pre-downloaded export ZIP or JSON.
|
||||
|
||||
Deprioritized 2026-06-12: the official ChatGPT export does not cover what
|
||||
this user needs (project data), so it isn't a real fallback here.
|
||||
|
||||
## Reclassify o1/o3 Reasoning Subparts
|
||||
|
||||
v0.4.0 leaves dict parts inside `text` content_type messages with shape
|
||||
`{"summary": ..., "content": ...}` rendered as plain text (defensive — the
|
||||
shape was inferred from a code comment, not captured live). Once a real
|
||||
reasoning conversation is captured, reclassify these as `thinking` blocks.
|
||||
|
||||
## Obsidian Vault Output
|
||||
|
||||
Add an `obsidian` command (or `--target obsidian` flag) to sync exported
|
||||
conversations into an Obsidian vault directory. The current Markdown format
|
||||
@@ -116,28 +371,7 @@ Obsidian. An `ObsidianSyncer` class (mirroring `JoplinClient`) would simply
|
||||
copy files to the vault directory and maintain a flat or nested folder
|
||||
structure matching the user's Obsidian setup. No API needed — just file I/O.
|
||||
|
||||
---
|
||||
|
||||
## Joplin Nested Notebooks (future)
|
||||
|
||||
Currently notebooks are flat: `ChatGPT - My Project`. Joplin supports nested
|
||||
notebooks via `parent_id`. A future option (`JOPLIN_NESTED_NOTEBOOKS=true`)
|
||||
could create a two-level hierarchy:
|
||||
|
||||
```
|
||||
ChatGPT/
|
||||
My Project/
|
||||
No Project/
|
||||
Claude/
|
||||
Budget Tracker/
|
||||
```
|
||||
|
||||
Implementation: `get_or_create_notebook` would first find/create the provider
|
||||
notebook, then find/create the project notebook as a child.
|
||||
|
||||
---
|
||||
|
||||
## Token Expiry Notifications (future)
|
||||
## Token Expiry Notifications
|
||||
|
||||
Proactively warn when a token is close to expiry (within 48h for ChatGPT),
|
||||
rather than only surfacing the warning at startup. Options:
|
||||
@@ -147,9 +381,10 @@ rather than only surfacing the warning at startup. Options:
|
||||
- Send a desktop notification via `notify-send` (Linux) or `osascript` (macOS)
|
||||
when a token is within 24h of expiry
|
||||
|
||||
---
|
||||
Largely superseded by Brave cookie auto-extraction (#2): once refresh is
|
||||
one command (or automatic), expiry warnings matter much less.
|
||||
|
||||
## Search Command (future)
|
||||
## Search Command
|
||||
|
||||
Add a `search` command to full-text search across all exported Markdown files:
|
||||
|
||||
|
||||
@@ -87,11 +87,22 @@ This checks token presence, format, expiry, directory permissions, disk space, a
|
||||
|
||||
Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
|
||||
|
||||
### The fast way: extract from your browser
|
||||
|
||||
If the browser you're logged in with runs on the same machine as this tool:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth --from-browser # Brave (default)
|
||||
ai-chat-exporter auth --from-browser firefox # or chrome, chromium, edge
|
||||
```
|
||||
|
||||
This reads the session cookies straight from the browser's cookie store (decrypted via the OS keyring/DPAPI/Keychain), validates each token against the live API, and writes `.env` — no DevTools, no copy-pasting. A token that fails validation never overwrites a working one. If extraction fails (browser not installed here, cookie DB locked, logged out), fall back to the manual flow below.
|
||||
|
||||
### Token Lifetimes
|
||||
|
||||
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
||||
|----------|-------------|----------|-----------------|
|
||||
| ChatGPT | `__Secure-next-auth.session-token` | ~7 days | JWT `exp` claim (decoded automatically) |
|
||||
| ChatGPT | `__Secure-next-auth.session-token.0` + `.1` | ~7 days | JWT `exp` claim (decoded automatically) |
|
||||
| Claude | `sessionKey` | ~30 days | Only detectable via 401 response |
|
||||
|
||||
### Finding Tokens in Chrome DevTools
|
||||
@@ -102,7 +113,11 @@ Session tokens are how your browser stays logged in. This tool uses them to acce
|
||||
4. In the left panel, expand **Cookies** and click the site URL
|
||||
5. Find the cookie by name and copy its **Value**
|
||||
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find `__Secure-next-auth.session-token` → copy Value (starts with `eyJ`)
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find **two** cookies:
|
||||
- `__Secure-next-auth.session-token.0` — copy Value (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`
|
||||
- `__Secure-next-auth.session-token.1` — copy Value → `CHATGPT_SESSION_TOKEN_1`
|
||||
|
||||
ChatGPT splits large session tokens across two cookies to stay under the browser's 4KB cookie limit. Both are required.
|
||||
|
||||
**Claude:** go to `https://claude.ai` → find `sessionKey` → copy Value
|
||||
|
||||
@@ -138,7 +153,8 @@ cp .env.example .env
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `CHATGPT_SESSION_TOKEN` | Your ChatGPT JWT session token (`eyJ…`) |
|
||||
| `CHATGPT_SESSION_TOKEN` | ChatGPT session token chunk `.0` (starts with `eyJ…`) |
|
||||
| `CHATGPT_SESSION_TOKEN_1` | ChatGPT session token chunk `.1` (the remainder) |
|
||||
| `CHATGPT_PROJECT_IDS` | Comma-separated ChatGPT project IDs (see below) |
|
||||
| `CLAUDE_SESSION_KEY` | Your Claude session key |
|
||||
|
||||
@@ -148,6 +164,10 @@ cp .env.example .env
|
||||
|----------|---------|-------------|
|
||||
| `EXPORT_DIR` | `./exports` | Where to write exported Markdown files |
|
||||
| `OUTPUT_STRUCTURE` | `provider/project/year` | Folder structure (see below) |
|
||||
| `EXPORTER_HIDDEN_CONTENT` | `placeholder` | What to do with content invisible in the provider's web UI (file-retrieval tool dumps, Custom Instructions): `placeholder` collapses to a one-line note with size, `full` keeps everything, `omit` drops it. Dumps can be 90% of a conversation's bytes. |
|
||||
| `EXPORTER_DOWNLOAD_MEDIA` | `images` | Download conversation assets into a `media/` folder beside each export and inline them: `images` (images only), `all` (also audio/voice clips), `off` (text placeholders only). Downloaded media is uploaded to Joplin as resources on the next `joplin` run. |
|
||||
| `MAX_CONVERSATIONS_PER_RUN` | unlimited | Cap downloads per export run (per provider). Runs are resumable, so re-running continues where the cap stopped — useful to spread a large first export across several sessions. |
|
||||
| `REQUEST_DELAY` | `1.0` | Seconds between consecutive API requests, with small random jitter, so traffic stays human-paced. Set `0` to disable. |
|
||||
|
||||
### Joplin
|
||||
|
||||
@@ -187,6 +207,19 @@ The `auth` wizard can also guide you through this step interactively.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Sessions
|
||||
|
||||
The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/` (override with `CLAUDE_CODE_DIR` in `.env`).
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider claude-code
|
||||
ai-chat-exporter joplin --provider claude-code
|
||||
```
|
||||
|
||||
Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Each coding project becomes a Joplin notebook under the `AI-Claude` parent. The provider is included in `--provider all` whenever the sessions directory exists.
|
||||
|
||||
---
|
||||
|
||||
## Output Structure
|
||||
|
||||
All exported files go under `EXPORT_DIR`. The folder structure maps directly to Joplin notebooks.
|
||||
@@ -251,10 +284,11 @@ Each provider+project combination maps to a flat Joplin notebook created automat
|
||||
### `auth` — Interactive token setup
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
ai-chat-exporter auth # manual wizard (DevTools flow)
|
||||
ai-chat-exporter auth --from-browser # extract from Brave's cookie store
|
||||
```
|
||||
|
||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut.
|
||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut. `--from-browser [brave|chrome|chromium|edge|firefox]` skips the wizard and pulls tokens directly from a local browser, validating them live before writing `.env`.
|
||||
|
||||
### `doctor` — Health check
|
||||
|
||||
@@ -295,7 +329,7 @@ ai-chat-exporter export --output /path/to/my/notes
|
||||
ai-chat-exporter export --dry-run
|
||||
```
|
||||
|
||||
Options: `--provider [chatgpt|claude|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--dry-run`
|
||||
Options: `--provider [chatgpt|claude|claude-code|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--dry-run`
|
||||
|
||||
### `list` — List conversations
|
||||
|
||||
@@ -344,6 +378,22 @@ Reads the local export cache and pushes each exported Markdown file to Joplin as
|
||||
|
||||
Options: `--provider [chatgpt|claude|all]`, `--project NAME`, `--dry-run`
|
||||
|
||||
### `prune` — Delete stale export files
|
||||
|
||||
```bash
|
||||
# Preview what would be deleted
|
||||
ai-chat-exporter prune --dry-run
|
||||
|
||||
# Delete (asks for confirmation; -y skips the prompt)
|
||||
ai-chat-exporter prune
|
||||
```
|
||||
|
||||
Deletes export files no longer referenced by the cache manifest — leftovers
|
||||
from old folder layouts or fixed bugs that would otherwise double-sync into
|
||||
Joplin. Refuses to run when the manifest is empty (e.g. right after
|
||||
`cache --clear`) so it can never wipe a freshly cleared archive. The `doctor`
|
||||
command separately verifies that every manifest entry's file exists on disk.
|
||||
|
||||
### `cache` — Manage the sync manifest
|
||||
|
||||
```bash
|
||||
@@ -421,7 +471,13 @@ Make sure you've added the project IDs to `CHATGPT_PROJECT_IDS` in your `.env`.
|
||||
The provider's internal API may have changed. Run with `--debug`, sanitize the output (remove any personal content), and check the project's GitHub Issues for known fixes.
|
||||
|
||||
### Non-text content warnings
|
||||
Images, code interpreter outputs, DALL-E generations, and Claude artifacts are not exported in v0.2.0. A WARNING is logged for each skipped item. See `FUTURE.md` for the roadmap.
|
||||
Since v0.4.0, rich content is preserved as typed blocks in the export. ChatGPT voice transcripts render as text and audio assets as `📎 File attached` placeholders with size and duration metadata. Anything the extractor doesn't recognise renders as a visible `> ⚠️ Unsupported content` block naming the type and observed keys, *and* increments a counter in the post-export summary so you can tell whether real content is being silently skipped.
|
||||
|
||||
### Embedded images and media
|
||||
Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as ``. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Assets that have expired on the provider's side (older AI-generated images often do) keep their text placeholder and are tallied as `media failed` in the run summary — they're never fatal to the export. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
|
||||
|
||||
### Tool output / hidden context collapsed
|
||||
Since v0.6.0, content that was invisible in the ChatGPT web UI is collapsed by default to one-line placeholders like `> 🔧 Tool output — file_search (15.0 KB) — omitted`. This is ChatGPT's file-retrieval tool re-injecting your attached files on every run — it can be 90% of a conversation's bytes while containing none of the dialogue. Custom Instructions collapse the same way (`> ℹ️ Hidden context`). The post-export summary lists every collapsed message by origin with the total KB omitted. To keep everything, set `EXPORTER_HIDDEN_CONTENT=full` in `.env` or pass `--hidden-content full`.
|
||||
|
||||
### Empty export / all conversations skipped
|
||||
No new or updated conversations since your last run. To verify: `ai-chat-exporter cache --show`. To force a full re-export: `ai-chat-exporter cache --clear`.
|
||||
@@ -435,12 +491,9 @@ No new or updated conversations since your last run. To verify: `ai-chat-exporte
|
||||
|
||||
## Future Work
|
||||
|
||||
See `FUTURE.md` for planned features:
|
||||
See `FUTURE.md` for the full roadmap. Current priorities:
|
||||
|
||||
- **v0.2.x** — `export --force` flag; `joplin --force` flag; per-conversation cache reset
|
||||
- **v0.3.0** — Official API fallback: parse export ZIP files from ChatGPT/Claude settings
|
||||
- **v0.4.0** — Rich content: images, artifacts, code interpreter output, extended thinking
|
||||
- **v0.5.0** — Watch/scheduled mode; Obsidian vault output
|
||||
- **Watch/scheduled mode** on the way to a headless StartOS service
|
||||
|
||||
---
|
||||
|
||||
|
||||
+2
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "ai-chat-exporter"
|
||||
version = "0.2.1"
|
||||
version = "0.6.0"
|
||||
description = "Export ChatGPT and Claude conversation history to Markdown for personal archival in Joplin"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
@@ -15,6 +15,7 @@ dependencies = [
|
||||
"rich==13.7.1",
|
||||
"python-slugify==8.0.4",
|
||||
"PyJWT==2.8.0",
|
||||
"browser-cookie3==0.20.1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
|
||||
@@ -26,3 +26,4 @@ setuptools==82.0.0
|
||||
text-unidecode==1.3
|
||||
urllib3==2.6.3
|
||||
wheel==0.46.3
|
||||
browser-cookie3==0.20.1
|
||||
|
||||
+399
@@ -0,0 +1,399 @@
|
||||
"""Typed content blocks for normalized messages.
|
||||
|
||||
Providers produce ordered lists of blocks; exporters render them. Living outside
|
||||
``src/providers/`` deliberately — blocks are a separate concern from extraction
|
||||
or rendering, shared by both layers.
|
||||
|
||||
Block dicts always have ``type`` set to one of the BLOCK_TYPE_* constants.
|
||||
Construct via the ``make_*`` helpers; never build dicts by hand. The ``unknown``
|
||||
block constructor REQUIRES a corresponding WARNING log + ``LossReport`` tally
|
||||
at the call site — see plan §Data-loss visibility.
|
||||
"""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
BLOCK_TYPE_TEXT = "text"
|
||||
BLOCK_TYPE_CODE = "code"
|
||||
BLOCK_TYPE_THINKING = "thinking"
|
||||
BLOCK_TYPE_TOOL_USE = "tool_use"
|
||||
BLOCK_TYPE_TOOL_RESULT = "tool_result"
|
||||
BLOCK_TYPE_CITATION = "citation"
|
||||
BLOCK_TYPE_IMAGE_PLACEHOLDER = "image_placeholder"
|
||||
BLOCK_TYPE_FILE_PLACEHOLDER = "file_placeholder"
|
||||
BLOCK_TYPE_UNKNOWN = "unknown"
|
||||
BLOCK_TYPE_HIDDEN_CONTEXT_MARKER = "hidden_context_marker"
|
||||
BLOCK_TYPE_COLLAPSED = "collapsed"
|
||||
|
||||
COLLAPSED_KIND_TOOL_DUMP = "tool_dump"
|
||||
COLLAPSED_KIND_HIDDEN_CONTEXT = "hidden_context"
|
||||
|
||||
UNKNOWN_REASON_UNKNOWN_TYPE = "unknown_type"
|
||||
UNKNOWN_REASON_EXTRACTION_FAILED = "extraction_failed"
|
||||
UNKNOWN_REASON_ALL_BLOCKS_FAILED = "all_blocks_failed"
|
||||
UNKNOWN_REASON_UNKNOWN_FIELD_IN_KNOWN_TYPE = "unknown_field_in_known_type"
|
||||
|
||||
_OBSERVED_KEYS_LIMIT = 10
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constructors
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def make_text_block(text: str) -> dict | None:
|
||||
"""Return a text block, or None if the text is empty/whitespace-only.
|
||||
|
||||
Returning None lets callers do ``if block: blocks.append(block)`` and prune
|
||||
empty blocks at construction time. See plan §Finalizing the message dict.
|
||||
"""
|
||||
if not isinstance(text, str) or not text.strip():
|
||||
return None
|
||||
return {"type": BLOCK_TYPE_TEXT, "text": text}
|
||||
|
||||
|
||||
def make_code_block(code: str, language: str = "") -> dict | None:
|
||||
"""Return a code block, or None if code is empty."""
|
||||
if not isinstance(code, str) or not code.strip():
|
||||
return None
|
||||
return {"type": BLOCK_TYPE_CODE, "language": language or "", "code": code}
|
||||
|
||||
|
||||
def make_thinking_block(text: str) -> dict | None:
|
||||
"""Return a thinking block, or None if empty."""
|
||||
if not isinstance(text, str) or not text.strip():
|
||||
return None
|
||||
return {"type": BLOCK_TYPE_THINKING, "text": text}
|
||||
|
||||
|
||||
def make_tool_use_block(name: str, input_data: Any, tool_id: str | None = None) -> dict:
|
||||
"""Return a tool_use block.
|
||||
|
||||
Always returns a block (no None) — tool calls are meaningful even with
|
||||
empty inputs.
|
||||
"""
|
||||
return {
|
||||
"type": BLOCK_TYPE_TOOL_USE,
|
||||
"name": name or "",
|
||||
"input": input_data if input_data is not None else {},
|
||||
"tool_id": tool_id,
|
||||
}
|
||||
|
||||
|
||||
def make_tool_result_block(
|
||||
output: str,
|
||||
tool_name: str | None = None,
|
||||
is_error: bool = False,
|
||||
summary: str | None = None,
|
||||
) -> dict:
|
||||
"""Return a tool_result block.
|
||||
|
||||
``summary`` is an optional short human label rendered between header and
|
||||
fence (e.g. ChatGPT's ``metadata.reasoning_title`` for execution_output).
|
||||
"""
|
||||
return {
|
||||
"type": BLOCK_TYPE_TOOL_RESULT,
|
||||
"tool_name": tool_name,
|
||||
"output": output if isinstance(output, str) else str(output),
|
||||
"is_error": bool(is_error),
|
||||
"summary": summary,
|
||||
}
|
||||
|
||||
|
||||
def make_citation_block(
|
||||
url: str,
|
||||
title: str | None = None,
|
||||
snippet: str | None = None,
|
||||
) -> dict | None:
|
||||
if not url:
|
||||
return None
|
||||
return {
|
||||
"type": BLOCK_TYPE_CITATION,
|
||||
"url": url,
|
||||
"title": title,
|
||||
"snippet": snippet,
|
||||
}
|
||||
|
||||
|
||||
def make_image_placeholder(
|
||||
ref: str,
|
||||
source: str = "unknown",
|
||||
mime: str | None = None,
|
||||
) -> dict:
|
||||
"""source ∈ {'user_upload', 'model_generated', 'unknown'}."""
|
||||
return {
|
||||
"type": BLOCK_TYPE_IMAGE_PLACEHOLDER,
|
||||
"ref": ref or "",
|
||||
"source": source,
|
||||
"mime": mime,
|
||||
}
|
||||
|
||||
|
||||
def make_file_placeholder(
|
||||
ref: str,
|
||||
filename: str | None = None,
|
||||
mime: str | None = None,
|
||||
size_bytes: int | None = None,
|
||||
duration_seconds: float | None = None,
|
||||
) -> dict:
|
||||
return {
|
||||
"type": BLOCK_TYPE_FILE_PLACEHOLDER,
|
||||
"ref": ref or "",
|
||||
"filename": filename,
|
||||
"mime": mime,
|
||||
"size_bytes": size_bytes,
|
||||
"duration_seconds": duration_seconds,
|
||||
}
|
||||
|
||||
|
||||
def make_unknown_block(
|
||||
raw_type: str,
|
||||
observed_keys: list[str] | None = None,
|
||||
reason: str = UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
summary: str | None = None,
|
||||
) -> dict:
|
||||
"""Construct an unknown block.
|
||||
|
||||
Every call site MUST also emit a WARNING log and increment a LossReport
|
||||
tally — see plan §Data-loss visibility. The block surfaces the loss at
|
||||
read time; the WARNING surfaces it at run time. Both signals matter.
|
||||
"""
|
||||
keys = list(observed_keys or [])[:_OBSERVED_KEYS_LIMIT]
|
||||
return {
|
||||
"type": BLOCK_TYPE_UNKNOWN,
|
||||
"raw_type": raw_type,
|
||||
"observed_keys": keys,
|
||||
"reason": reason,
|
||||
"summary": summary,
|
||||
}
|
||||
|
||||
|
||||
def make_collapsed_block(
|
||||
origin: str,
|
||||
content_type: str,
|
||||
size_bytes: int,
|
||||
kind: str,
|
||||
) -> dict:
|
||||
"""One-line stand-in for a message omitted by the EXPORTER_HIDDEN_CONTENT policy.
|
||||
|
||||
``kind`` ∈ {COLLAPSED_KIND_TOOL_DUMP, COLLAPSED_KIND_HIDDEN_CONTEXT}.
|
||||
Unlike ``unknown`` blocks (unexpected loss), a collapsed block records an
|
||||
intentional policy decision — the content was invisible in the provider's
|
||||
web UI and the user chose not to keep it. Every call site MUST tally via
|
||||
``LossReport.record_collapsed`` so the omission stays visible in the
|
||||
post-export summary.
|
||||
"""
|
||||
return {
|
||||
"type": BLOCK_TYPE_COLLAPSED,
|
||||
"origin": origin or "",
|
||||
"content_type": content_type or "",
|
||||
"size_bytes": int(size_bytes or 0),
|
||||
"kind": kind,
|
||||
}
|
||||
|
||||
|
||||
def make_hidden_context_marker(content_type: str) -> dict:
|
||||
"""A short prepend block that flags the surrounding message as hidden context.
|
||||
|
||||
Driven by the ``metadata.is_visually_hidden_from_conversation`` flag, not by
|
||||
content_type matching. The marker tells the reader "this message is
|
||||
hidden in the source UI; we're showing it here for archival fidelity."
|
||||
"""
|
||||
return {
|
||||
"type": BLOCK_TYPE_HIDDEN_CONTEXT_MARKER,
|
||||
"content_type": content_type or "",
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rendering
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def render_blocks_to_markdown(blocks: list[dict]) -> str:
|
||||
"""Render an ordered list of blocks to a single Markdown string.
|
||||
|
||||
Blocks are joined with one blank line between them. Pure function; no I/O.
|
||||
"""
|
||||
if not blocks:
|
||||
return ""
|
||||
|
||||
rendered: list[str] = []
|
||||
for block in blocks:
|
||||
chunk = _render_one(block)
|
||||
if chunk:
|
||||
rendered.append(chunk)
|
||||
|
||||
return "\n\n".join(rendered)
|
||||
|
||||
|
||||
def _render_one(block: dict) -> str:
|
||||
btype = block.get("type", "")
|
||||
if btype == BLOCK_TYPE_TEXT:
|
||||
return block.get("text", "")
|
||||
if btype == BLOCK_TYPE_CODE:
|
||||
lang = block.get("language") or ""
|
||||
code = block.get("code", "")
|
||||
fence = _safe_fence(code)
|
||||
return f"{fence}{lang}\n{code}\n{fence}"
|
||||
if btype == BLOCK_TYPE_THINKING:
|
||||
text = block.get("text", "")
|
||||
quoted = _blockquote_prefix(text)
|
||||
return f"**💭 Reasoning**\n\n{quoted}"
|
||||
if btype == BLOCK_TYPE_TOOL_USE:
|
||||
name = block.get("name", "")
|
||||
input_data = block.get("input", {})
|
||||
body_json = json.dumps(input_data, indent=2, sort_keys=False, default=str, ensure_ascii=False)
|
||||
fence = _safe_fence(body_json)
|
||||
body = f"{fence}json\n{body_json}\n{fence}"
|
||||
quoted = _blockquote_prefix(f"🔧 **Tool: {name}**\n{body}")
|
||||
return quoted
|
||||
if btype == BLOCK_TYPE_TOOL_RESULT:
|
||||
output = block.get("output", "")
|
||||
is_error = bool(block.get("is_error"))
|
||||
tool_name = block.get("tool_name") or ""
|
||||
summary = block.get("summary") or ""
|
||||
icon = "❌" if is_error else "📤"
|
||||
label = "Result (error)" if is_error else "Result"
|
||||
if tool_name:
|
||||
header = f"{icon} **{label}: {tool_name}**"
|
||||
else:
|
||||
header = f"{icon} **{label}**"
|
||||
fence = _safe_fence(output)
|
||||
body = f"{fence}\n{output}\n{fence}"
|
||||
if summary:
|
||||
inner = f"{header}\n*{summary}*\n{body}"
|
||||
else:
|
||||
inner = f"{header}\n{body}"
|
||||
return _blockquote_prefix(inner)
|
||||
if btype == BLOCK_TYPE_CITATION:
|
||||
url = block.get("url", "")
|
||||
title = block.get("title") or url
|
||||
return f"[{title}]({url})"
|
||||
if btype == BLOCK_TYPE_IMAGE_PLACEHOLDER:
|
||||
ref = block.get("ref", "")
|
||||
source = block.get("source", "unknown")
|
||||
mime = block.get("mime")
|
||||
local_path = block.get("local_path")
|
||||
if local_path:
|
||||
# Downloaded — render as a real inline image.
|
||||
return f""
|
||||
meta_parts = [source] if source else []
|
||||
if mime:
|
||||
meta_parts.append(mime)
|
||||
meta_parts.append("content not preserved in this export")
|
||||
meta = ", ".join(meta_parts)
|
||||
return f"> 🖼️ **Image attached** — `{ref}` ({meta})"
|
||||
if btype == BLOCK_TYPE_FILE_PLACEHOLDER:
|
||||
ref = block.get("ref", "")
|
||||
filename = block.get("filename")
|
||||
label = filename or ref
|
||||
mime = block.get("mime")
|
||||
size_bytes = block.get("size_bytes")
|
||||
duration = block.get("duration_seconds")
|
||||
local_path = block.get("local_path")
|
||||
meta_parts: list[str] = []
|
||||
if mime:
|
||||
meta_parts.append(mime)
|
||||
size_label = _format_size(size_bytes)
|
||||
if size_label:
|
||||
meta_parts.append(size_label)
|
||||
if isinstance(duration, (int, float)) and duration > 0:
|
||||
meta_parts.append(f"{duration:.2f}s")
|
||||
if local_path:
|
||||
# Downloaded — render as a link to the local copy.
|
||||
meta = ", ".join(meta_parts) if meta_parts else ""
|
||||
meta_str = f" ({meta})" if meta else ""
|
||||
return f"> 📎 **File attached** — [{Path(local_path).name}]({local_path}){meta_str}"
|
||||
meta_parts.append("content not preserved in this export")
|
||||
meta = ", ".join(meta_parts)
|
||||
return f"> 📎 **File attached** — `{label}` ({meta})"
|
||||
if btype == BLOCK_TYPE_UNKNOWN:
|
||||
raw_type = block.get("raw_type", "?")
|
||||
reason = block.get("reason", UNKNOWN_REASON_UNKNOWN_TYPE)
|
||||
keys = block.get("observed_keys") or []
|
||||
summary = block.get("summary")
|
||||
first_line = f"⚠️ **Unsupported content** — type `{raw_type}` ({reason})"
|
||||
lines = [first_line]
|
||||
if summary:
|
||||
lines.append(summary)
|
||||
if keys:
|
||||
keys_str = ", ".join(f"`{k}`" for k in keys)
|
||||
lines.append(f"Keys observed: {keys_str}")
|
||||
return _blockquote_prefix("\n".join(lines))
|
||||
if btype == BLOCK_TYPE_HIDDEN_CONTEXT_MARKER:
|
||||
ctype = block.get("content_type", "")
|
||||
return f"> ℹ️ **Hidden context** — `{ctype}`"
|
||||
if btype == BLOCK_TYPE_COLLAPSED:
|
||||
origin = block.get("origin", "")
|
||||
kind = block.get("kind", "")
|
||||
if kind == COLLAPSED_KIND_HIDDEN_CONTEXT:
|
||||
icon, label = "ℹ️", "Hidden context"
|
||||
else:
|
||||
icon, label = "🔧", "Tool output"
|
||||
size_label = _format_size(block.get("size_bytes"))
|
||||
size_part = f" ({size_label})" if size_label else ""
|
||||
return (
|
||||
f"> {icon} **{label}** — `{origin}`{size_part} — omitted "
|
||||
"(EXPORTER_HIDDEN_CONTENT=full to keep)"
|
||||
)
|
||||
|
||||
# Defensive: a block of unrecognised local type (shouldn't happen if
|
||||
# constructors are used). Render as visible warning rather than dropping.
|
||||
return f"> ⚠️ **Internal: unrecognised block type** — `{btype}`"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _format_size(size_bytes: Any) -> str:
|
||||
"""Human-readable size ('24.1 KB', '1.05 MB'), or '' when absent/zero."""
|
||||
if not isinstance(size_bytes, int) or size_bytes <= 0:
|
||||
return ""
|
||||
kb = size_bytes / 1024
|
||||
return f"{kb:.1f} KB" if kb < 1024 else f"{kb / 1024:.2f} MB"
|
||||
|
||||
|
||||
def _safe_fence(text: str) -> str:
|
||||
"""Return a backtick fence longer than the longest run of backticks in ``text``.
|
||||
|
||||
CommonMark requires the closing fence to be at least as long as the opening
|
||||
fence. Picking N+1 (where N = longest run in content) ensures the content's
|
||||
own backticks are inert. Minimum is 3.
|
||||
|
||||
Verified live against Joplin during planning — see plan
|
||||
§Backtick-corruption defense.
|
||||
"""
|
||||
if not isinstance(text, str):
|
||||
return "```"
|
||||
longest_run = 0
|
||||
current_run = 0
|
||||
for ch in text:
|
||||
if ch == "`":
|
||||
current_run += 1
|
||||
if current_run > longest_run:
|
||||
longest_run = current_run
|
||||
else:
|
||||
current_run = 0
|
||||
fence_len = max(3, longest_run + 1)
|
||||
return "`" * fence_len
|
||||
|
||||
|
||||
def _blockquote_prefix(text: str) -> str:
|
||||
"""Prefix every line of ``text`` with ``> `` so the whole block renders as a quote.
|
||||
|
||||
Empty source lines become ``>`` (no trailing space) so blockquote continuity
|
||||
is preserved without trailing-whitespace noise.
|
||||
"""
|
||||
if not isinstance(text, str):
|
||||
return ""
|
||||
out_lines: list[str] = []
|
||||
for line in text.split("\n"):
|
||||
if line == "":
|
||||
out_lines.append(">")
|
||||
else:
|
||||
out_lines.append(f"> {line}")
|
||||
return "\n".join(out_lines)
|
||||
@@ -0,0 +1,101 @@
|
||||
"""Extract session tokens directly from a local browser's cookie store.
|
||||
|
||||
Default browser is Brave. Uses browser-cookie3, which handles the
|
||||
platform-specific cookie decryption (Linux: AES key derived from the OS
|
||||
keyring "Safe Storage" secret; Windows: DPAPI; macOS: Keychain). The browser
|
||||
must be the one the user is actually logged in with, on the same machine.
|
||||
|
||||
Failure modes worth knowing:
|
||||
- Browser not installed / profile not found → BrowserTokenError
|
||||
- Cookie DB locked (browser running with exclusive lock; rare on Linux,
|
||||
common on Windows) → BrowserTokenError suggesting the browser be closed
|
||||
- Logged out / cookie expired → BrowserTokenError naming the missing cookie
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_BROWSER = "brave"
|
||||
SUPPORTED_BROWSERS = ("brave", "chrome", "chromium", "edge", "firefox")
|
||||
|
||||
# ChatGPT splits large session tokens across two cookies to stay under the
|
||||
# browser's 4KB cookie limit; small tokens use the unsuffixed name.
|
||||
_CHATGPT_COOKIE_0 = "__Secure-next-auth.session-token.0"
|
||||
_CHATGPT_COOKIE_1 = "__Secure-next-auth.session-token.1"
|
||||
_CHATGPT_COOKIE_SINGLE = "__Secure-next-auth.session-token"
|
||||
_CLAUDE_COOKIE = "sessionKey"
|
||||
|
||||
|
||||
class BrowserTokenError(Exception):
|
||||
"""Raised when tokens cannot be extracted from the browser profile."""
|
||||
|
||||
|
||||
def _load_cookies(browser: str, domain: str) -> dict[str, str]:
|
||||
"""Return {cookie_name: value} for ``domain`` from the given browser."""
|
||||
if browser not in SUPPORTED_BROWSERS:
|
||||
raise BrowserTokenError(
|
||||
f"Unsupported browser {browser!r}. "
|
||||
f"Supported: {', '.join(SUPPORTED_BROWSERS)}"
|
||||
)
|
||||
|
||||
import browser_cookie3
|
||||
|
||||
loader = getattr(browser_cookie3, browser)
|
||||
try:
|
||||
jar = loader(domain_name=domain)
|
||||
except Exception as e:
|
||||
# browser_cookie3 raises BrowserCookieError plus assorted OS/keyring
|
||||
# errors. All mean the same thing to the caller: fall back to manual.
|
||||
hint = ""
|
||||
text = str(e).lower()
|
||||
if "lock" in text or "database is locked" in text:
|
||||
hint = " Close the browser and try again."
|
||||
elif "not find" in text or "no such file" in text:
|
||||
hint = " Is the browser installed on this machine?"
|
||||
raise BrowserTokenError(
|
||||
f"Could not read {browser} cookies for {domain}: {e}.{hint}"
|
||||
) from e
|
||||
|
||||
return {cookie.name: cookie.value or "" for cookie in jar}
|
||||
|
||||
|
||||
def extract_chatgpt_tokens(browser: str = DEFAULT_BROWSER) -> tuple[str, str | None]:
|
||||
"""Return (session_token_0, session_token_1_or_None) from the browser.
|
||||
|
||||
Raises BrowserTokenError when no session cookie is present (not logged in,
|
||||
or logged out since the last visit).
|
||||
"""
|
||||
cookies = _load_cookies(browser, "chatgpt.com")
|
||||
token_0 = cookies.get(_CHATGPT_COOKIE_0, "").strip()
|
||||
token_1 = cookies.get(_CHATGPT_COOKIE_1, "").strip() or None
|
||||
if token_0:
|
||||
logger.info(
|
||||
"[browser-tokens] ChatGPT session cookies found in %s (chunked: %s)",
|
||||
browser,
|
||||
"yes" if token_1 else "no",
|
||||
)
|
||||
return token_0, token_1
|
||||
|
||||
single = cookies.get(_CHATGPT_COOKIE_SINGLE, "").strip()
|
||||
if single:
|
||||
logger.info("[browser-tokens] ChatGPT session cookie found in %s (single)", browser)
|
||||
return single, None
|
||||
|
||||
raise BrowserTokenError(
|
||||
f"No ChatGPT session cookie in {browser} — open https://chatgpt.com "
|
||||
"in that browser and log in, then retry."
|
||||
)
|
||||
|
||||
|
||||
def extract_claude_session(browser: str = DEFAULT_BROWSER) -> str:
|
||||
"""Return the Claude ``sessionKey`` cookie value from the browser."""
|
||||
cookies = _load_cookies(browser, "claude.ai")
|
||||
session_key = cookies.get(_CLAUDE_COOKIE, "").strip()
|
||||
if session_key:
|
||||
logger.info("[browser-tokens] Claude sessionKey found in %s", browser)
|
||||
return session_key
|
||||
raise BrowserTokenError(
|
||||
f"No Claude sessionKey cookie in {browser} — open https://claude.ai "
|
||||
"in that browser and log in, then retry."
|
||||
)
|
||||
@@ -87,6 +87,7 @@ class Cache:
|
||||
self._data[provider][conv_id] = {
|
||||
"title": metadata.get("title", ""),
|
||||
"project": metadata.get("project"),
|
||||
"created_at": metadata.get("created_at", ""),
|
||||
"updated_at": metadata.get("updated_at", ""),
|
||||
"exported_at": datetime.now(tz=timezone.utc).isoformat(),
|
||||
"file_path": metadata.get("file_path", ""),
|
||||
@@ -172,6 +173,22 @@ class Cache:
|
||||
entry["joplin_synced_at"] = datetime.now(tz=timezone.utc).isoformat()
|
||||
self._save()
|
||||
|
||||
def get_joplin_resources(self, provider: str, conv_id: str) -> dict[str, str]:
|
||||
"""Return the {relative_media_path: joplin_resource_id} map for a conversation."""
|
||||
entry = self._data.get(provider, {}).get(conv_id)
|
||||
if not isinstance(entry, dict):
|
||||
return {}
|
||||
resources = entry.get("joplin_resources")
|
||||
return dict(resources) if isinstance(resources, dict) else {}
|
||||
|
||||
def set_joplin_resources(self, provider: str, conv_id: str, resources: dict[str, str]) -> None:
|
||||
"""Persist the {relative_media_path: joplin_resource_id} map for a conversation."""
|
||||
entry = self._data.get(provider, {}).get(conv_id)
|
||||
if entry is None:
|
||||
return
|
||||
entry["joplin_resources"] = dict(resources)
|
||||
self._save()
|
||||
|
||||
def get_joplin_pending(self, provider: str) -> list[tuple[str, dict]]:
|
||||
"""Return (conv_id, entry) pairs that need to be synced to Joplin.
|
||||
|
||||
@@ -209,6 +226,25 @@ class Cache:
|
||||
|
||||
return pending
|
||||
|
||||
def all_file_paths(self) -> set[str]:
|
||||
"""Return every non-empty ``file_path`` in the manifest, across providers.
|
||||
|
||||
Used by ``prune`` (files on disk but not here are stale) and by the
|
||||
doctor integrity check (files here but not on disk are missing).
|
||||
"""
|
||||
paths: set[str] = set()
|
||||
for key, entries in self._data.items():
|
||||
if not isinstance(entries, dict) or key in (
|
||||
"version",
|
||||
"last_run",
|
||||
"tos_acknowledged_at",
|
||||
):
|
||||
continue
|
||||
for entry in entries.values():
|
||||
if isinstance(entry, dict) and entry.get("file_path"):
|
||||
paths.add(entry["file_path"])
|
||||
return paths
|
||||
|
||||
def last_run(self) -> str | None:
|
||||
"""Return the ISO8601 timestamp of the last export run, or None."""
|
||||
return self._data.get("last_run")
|
||||
|
||||
+75
-1
@@ -20,6 +20,12 @@ _CLAUDE_PLACEHOLDER = ""
|
||||
# Valid OUTPUT_STRUCTURE values
|
||||
VALID_STRUCTURES = {"provider/project/year", "provider/project", "provider/year"}
|
||||
|
||||
# Valid EXPORTER_HIDDEN_CONTENT values (see src/providers/base.py)
|
||||
VALID_HIDDEN_CONTENT = {"full", "placeholder", "omit"}
|
||||
|
||||
# Valid EXPORTER_DOWNLOAD_MEDIA values (see src/media.py)
|
||||
VALID_DOWNLOAD_MEDIA = {"images", "all", "off"}
|
||||
|
||||
|
||||
class ConfigError(Exception):
|
||||
"""Raised when required configuration is missing or invalid."""
|
||||
@@ -43,6 +49,16 @@ class Config:
|
||||
# Joplin local REST API settings (Web Clipper service)
|
||||
joplin_api_token: str | None = None
|
||||
joplin_api_url: str = "http://localhost:41184"
|
||||
# Policy for content invisible in the provider web UI (retrieval dumps,
|
||||
# hidden context): full | placeholder | omit
|
||||
hidden_content: str = "placeholder"
|
||||
# Session cap: max conversations downloaded per export run (None = unlimited).
|
||||
# Every run is resumable, so a capped run just continues next time.
|
||||
max_conversations: int | None = None
|
||||
# Seconds between consecutive API requests (politeness pacing; 0 disables)
|
||||
request_delay: float = 1.0
|
||||
# Asset download policy: images | all | off
|
||||
download_media: str = "images"
|
||||
|
||||
|
||||
def load_config() -> Config:
|
||||
@@ -67,6 +83,12 @@ def load_config() -> Config:
|
||||
joplin_token = os.getenv("JOPLIN_API_TOKEN", "").strip() or None
|
||||
joplin_url = os.getenv("JOPLIN_API_URL", "http://localhost:41184").strip()
|
||||
|
||||
hidden_content = os.getenv("EXPORTER_HIDDEN_CONTENT", "").strip().lower() or "placeholder"
|
||||
|
||||
max_conversations_raw = os.getenv("MAX_CONVERSATIONS_PER_RUN", "").strip()
|
||||
request_delay_raw = os.getenv("REQUEST_DELAY", "").strip()
|
||||
download_media = os.getenv("EXPORTER_DOWNLOAD_MEDIA", "").strip().lower() or "images"
|
||||
|
||||
# Parse CHATGPT_PROJECT_IDS — comma-separated list of gizmo IDs (g-p-xxx)
|
||||
_project_ids_raw = os.getenv("CHATGPT_PROJECT_IDS", "").strip()
|
||||
chatgpt_project_ids = [
|
||||
@@ -90,6 +112,46 @@ def load_config() -> Config:
|
||||
f"Must be one of: {', '.join(sorted(VALID_STRUCTURES))}"
|
||||
)
|
||||
|
||||
# Validate hidden-content policy
|
||||
if hidden_content not in VALID_HIDDEN_CONTENT:
|
||||
errors.append(
|
||||
f"EXPORTER_HIDDEN_CONTENT '{hidden_content}' is invalid. "
|
||||
f"Must be one of: {', '.join(sorted(VALID_HIDDEN_CONTENT))}"
|
||||
)
|
||||
|
||||
# Validate media download policy
|
||||
if download_media not in VALID_DOWNLOAD_MEDIA:
|
||||
errors.append(
|
||||
f"EXPORTER_DOWNLOAD_MEDIA '{download_media}' is invalid. "
|
||||
f"Must be one of: {', '.join(sorted(VALID_DOWNLOAD_MEDIA))}"
|
||||
)
|
||||
|
||||
# Validate session cap
|
||||
max_conversations: int | None = None
|
||||
if max_conversations_raw:
|
||||
try:
|
||||
max_conversations = int(max_conversations_raw)
|
||||
except ValueError:
|
||||
errors.append(
|
||||
f"MAX_CONVERSATIONS_PER_RUN '{max_conversations_raw}' is not an integer."
|
||||
)
|
||||
else:
|
||||
if max_conversations < 1:
|
||||
errors.append(
|
||||
f"MAX_CONVERSATIONS_PER_RUN must be at least 1 (got {max_conversations})."
|
||||
)
|
||||
|
||||
# Validate request pacing
|
||||
request_delay = 1.0
|
||||
if request_delay_raw:
|
||||
try:
|
||||
request_delay = float(request_delay_raw)
|
||||
except ValueError:
|
||||
errors.append(f"REQUEST_DELAY '{request_delay_raw}' is not a number.")
|
||||
else:
|
||||
if request_delay < 0:
|
||||
errors.append(f"REQUEST_DELAY must be >= 0 (got {request_delay}).")
|
||||
|
||||
# Validate and decode ChatGPT JWT
|
||||
chatgpt_expiry: datetime | None = None
|
||||
if chatgpt_token:
|
||||
@@ -139,6 +201,10 @@ def load_config() -> Config:
|
||||
chatgpt_project_ids=chatgpt_project_ids,
|
||||
joplin_api_token=joplin_token,
|
||||
joplin_api_url=joplin_url,
|
||||
hidden_content=hidden_content,
|
||||
max_conversations=max_conversations,
|
||||
request_delay=request_delay,
|
||||
download_media=download_media,
|
||||
)
|
||||
|
||||
_log_startup_summary(config)
|
||||
@@ -223,7 +289,11 @@ def _log_startup_summary(cfg: Config) -> None:
|
||||
"Joplin: %s | "
|
||||
"export_dir=%s | "
|
||||
"structure=%s | "
|
||||
"cache_dir=%s",
|
||||
"cache_dir=%s | "
|
||||
"hidden_content=%s | "
|
||||
"max_conversations=%s | "
|
||||
"request_delay=%.1fs | "
|
||||
"download_media=%s",
|
||||
chatgpt_status,
|
||||
claude_status,
|
||||
len(cfg.chatgpt_project_ids),
|
||||
@@ -231,4 +301,8 @@ def _log_startup_summary(cfg: Config) -> None:
|
||||
cfg.export_dir,
|
||||
cfg.output_structure,
|
||||
cfg.cache_dir,
|
||||
cfg.hidden_content,
|
||||
cfg.max_conversations if cfg.max_conversations is not None else "unlimited",
|
||||
cfg.request_delay,
|
||||
cfg.download_media,
|
||||
)
|
||||
|
||||
@@ -6,6 +6,7 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from src.blocks import render_blocks_to_markdown
|
||||
from src.utils import build_export_path, generate_filename
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -15,6 +16,7 @@ _ROLE_LABELS = {
|
||||
"user": ("🧑 Human", "user"),
|
||||
"assistant": ("🤖 Assistant", "assistant"),
|
||||
"system": ("⚙️ System", "system"),
|
||||
"tool": ("🔧 Tool", "tool"),
|
||||
}
|
||||
|
||||
|
||||
@@ -125,10 +127,17 @@ class MarkdownExporter:
|
||||
# Messages
|
||||
for msg in messages:
|
||||
role = msg.get("role", "user")
|
||||
content = msg.get("content", "")
|
||||
blocks = msg.get("blocks") or []
|
||||
timestamp = msg.get("timestamp")
|
||||
|
||||
if not content or not content.strip():
|
||||
# Prefer rendering from blocks (v0.4.0+). Backward-compat fallback:
|
||||
# if blocks is missing/empty AND content exists, render content as-is.
|
||||
if blocks:
|
||||
body = render_blocks_to_markdown(blocks)
|
||||
else:
|
||||
body = msg.get("content", "") or ""
|
||||
|
||||
if not body or not body.strip():
|
||||
logger.warning(
|
||||
"[markdown] Skipping empty/whitespace message in conversation %s",
|
||||
conv_id[:8],
|
||||
@@ -143,7 +152,7 @@ class MarkdownExporter:
|
||||
else:
|
||||
lines.append("")
|
||||
|
||||
lines.append(content)
|
||||
lines.append(body)
|
||||
lines.append("")
|
||||
lines.append("---")
|
||||
lines.append("")
|
||||
|
||||
+139
-26
@@ -1,7 +1,10 @@
|
||||
"""Joplin Data API client for importing notes into Joplin desktop."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import requests
|
||||
@@ -32,8 +35,8 @@ class JoplinClient:
|
||||
def __init__(self, base_url: str, token: str) -> None:
|
||||
self._base_url = base_url.rstrip("/")
|
||||
self._token = token
|
||||
# In-memory cache of notebook title → ID to avoid repeated GET /folders
|
||||
self._notebook_cache: dict[str, str] = {}
|
||||
# In-memory cache: (parent_id | None, title) → folder ID
|
||||
self._notebook_cache: dict[tuple[str | None, str], str] = {}
|
||||
self._notebooks_loaded = False
|
||||
logger.debug("[joplin] Client initialised with base_url=%s", self._base_url)
|
||||
|
||||
@@ -89,13 +92,13 @@ class JoplinClient:
|
||||
"""Return all Joplin notebooks (folders), handling pagination.
|
||||
|
||||
Returns:
|
||||
List of folder dicts with at least ``id`` and ``title`` keys.
|
||||
List of folder dicts with at least ``id``, ``title``, and ``parent_id`` keys.
|
||||
"""
|
||||
results: list[dict] = []
|
||||
page = 1
|
||||
while True:
|
||||
logger.debug("[joplin] GET /folders page=%d", page)
|
||||
resp = self._get("/folders", params={"page": page, "fields": "id,title"})
|
||||
resp = self._get("/folders", params={"page": page, "fields": "id,title,parent_id"})
|
||||
items = resp.get("items", [])
|
||||
results.extend(items)
|
||||
logger.debug("[joplin] /folders page=%d → %d items, has_more=%s", page, len(items), resp.get("has_more"))
|
||||
@@ -104,11 +107,12 @@ class JoplinClient:
|
||||
page += 1
|
||||
return results
|
||||
|
||||
def get_or_create_notebook(self, title: str) -> str:
|
||||
"""Return the Joplin folder ID for ``title``, creating it if needed.
|
||||
def get_or_create_notebook(self, title: str, parent_id: str | None = None) -> str:
|
||||
"""Return the Joplin folder ID for ``title`` under ``parent_id``, creating if needed.
|
||||
|
||||
Args:
|
||||
title: Notebook display name (e.g. "ChatGPT - My Project").
|
||||
title: Notebook display name.
|
||||
parent_id: ID of the parent folder, or None for a root notebook.
|
||||
|
||||
Returns:
|
||||
Joplin folder ID string.
|
||||
@@ -116,19 +120,40 @@ class JoplinClient:
|
||||
if not self._notebooks_loaded:
|
||||
self._load_notebook_cache()
|
||||
|
||||
if title in self._notebook_cache:
|
||||
folder_id = self._notebook_cache[title]
|
||||
logger.debug("[joplin] Notebook cache hit: %r → %s", title, folder_id)
|
||||
key = (parent_id, title)
|
||||
if key in self._notebook_cache:
|
||||
folder_id = self._notebook_cache[key]
|
||||
logger.debug("[joplin] Notebook cache hit: %r (parent=%s) → %s", title, parent_id, folder_id)
|
||||
return folder_id
|
||||
|
||||
# Not found — create it
|
||||
logger.info("[joplin] Creating notebook: %r", title)
|
||||
resp = self._post("/folders", {"title": title})
|
||||
logger.info("[joplin] Creating notebook: %r (parent=%s)", title, parent_id)
|
||||
data: dict = {"title": title}
|
||||
if parent_id:
|
||||
data["parent_id"] = parent_id
|
||||
resp = self._post("/folders", data)
|
||||
folder_id = resp["id"]
|
||||
self._notebook_cache[title] = folder_id
|
||||
self._notebook_cache[key] = folder_id
|
||||
logger.debug("[joplin] Notebook created: %r → %s", title, folder_id)
|
||||
return folder_id
|
||||
|
||||
def get_or_create_notebook_path(self, path: list[str]) -> str:
|
||||
"""Ensure a nested notebook path exists and return the leaf folder ID.
|
||||
|
||||
Creates intermediate notebooks as needed.
|
||||
|
||||
Args:
|
||||
path: Ordered list of notebook names, e.g. ["AI-ChatGPT", "No Project"].
|
||||
|
||||
Returns:
|
||||
Joplin folder ID of the deepest (leaf) notebook.
|
||||
"""
|
||||
parent_id: str | None = None
|
||||
for name in path:
|
||||
parent_id = self.get_or_create_notebook(name, parent_id)
|
||||
assert parent_id is not None
|
||||
return parent_id
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Notes
|
||||
# ------------------------------------------------------------------
|
||||
@@ -168,6 +193,46 @@ class JoplinClient:
|
||||
self._put(f"/notes/{note_id}", {"title": title, "body": body})
|
||||
logger.info("[joplin] Note updated: %r (%s)", title, note_id)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Resources (embedded media)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def create_resource(self, file_path: "Path", title: str | None = None) -> str:
|
||||
"""Upload a local file as a Joplin resource and return its ID.
|
||||
|
||||
Joplin's ``POST /resources`` is multipart: a ``data`` file part plus a
|
||||
``props`` JSON part. The returned ID is referenced from note bodies as
|
||||
``:/<id>``.
|
||||
"""
|
||||
url = f"{self._base_url}/resources"
|
||||
props = json.dumps({"title": title or file_path.name})
|
||||
logger.debug("[joplin] POST /resources (%s)", file_path.name)
|
||||
try:
|
||||
with file_path.open("rb") as fh:
|
||||
resp = requests.post(
|
||||
url,
|
||||
params={"token": self._token},
|
||||
files={
|
||||
"data": (file_path.name, fh),
|
||||
"props": (None, props),
|
||||
},
|
||||
timeout=_REQUEST_TIMEOUT,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
resource_id = resp.json()["id"]
|
||||
logger.info("[joplin] Resource created: %s → %s", file_path.name, resource_id)
|
||||
return resource_id
|
||||
except requests.exceptions.ConnectionError as e:
|
||||
raise JoplinError(
|
||||
"Cannot connect to Joplin. Is Joplin desktop running with Web Clipper enabled?"
|
||||
) from e
|
||||
except requests.exceptions.Timeout as e:
|
||||
raise JoplinError(_timeout_message("POST", "/resources")) from e
|
||||
except requests.exceptions.HTTPError as e:
|
||||
raise JoplinError(_http_error_message("POST", "/resources", e)) from e
|
||||
except (requests.exceptions.RequestException, KeyError, OSError) as e:
|
||||
raise JoplinError(f"Joplin resource upload failed for {file_path.name}: {e}") from e
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# HTTP helpers
|
||||
# ------------------------------------------------------------------
|
||||
@@ -233,11 +298,14 @@ class JoplinClient:
|
||||
def _load_notebook_cache(self) -> None:
|
||||
logger.debug("[joplin] Loading notebook list from Joplin…")
|
||||
notebooks = self.list_notebooks()
|
||||
self._notebook_cache = {nb["title"]: nb["id"] for nb in notebooks}
|
||||
self._notebook_cache = {
|
||||
(nb.get("parent_id") or None, nb["title"]): nb["id"]
|
||||
for nb in notebooks
|
||||
}
|
||||
self._notebooks_loaded = True
|
||||
logger.debug("[joplin] Notebook cache loaded: %d notebooks", len(self._notebook_cache))
|
||||
for title, folder_id in self._notebook_cache.items():
|
||||
logger.debug("[joplin] %r → %s", title, folder_id)
|
||||
for (parent_id, title), folder_id in self._notebook_cache.items():
|
||||
logger.debug("[joplin] (%s) %r → %s", parent_id or "root", title, folder_id)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -279,25 +347,70 @@ def _http_error_message(method: str, path: str, e: requests.exceptions.HTTPError
|
||||
return f"Joplin {method} {path} failed: HTTP {status}{body_snippet}"
|
||||
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Embedded-media rewriting
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
# Markdown image/link targets pointing at the local media/ sibling directory,
|
||||
# e.g.  or [name](media/clip.wav).
|
||||
_MEDIA_LINK_RE = re.compile(r"(!?\[[^\]]*\])\((media/[^)\s]+)\)")
|
||||
|
||||
|
||||
def upload_media_and_rewrite(
|
||||
body: str,
|
||||
note_dir: Path,
|
||||
client: "JoplinClient",
|
||||
resource_map: dict[str, str],
|
||||
) -> tuple[str, dict[str, str]]:
|
||||
"""Rewrite local ``media/…`` links to Joplin ``:/resourceId`` references.
|
||||
|
||||
Uploads each referenced file as a Joplin resource (once), rewriting both
|
||||
image embeds and file links. ``resource_map`` (relative-path → resource_id)
|
||||
is the idempotency record from the cache: known paths are reused, new ones
|
||||
are added and returned so the caller can persist them. Missing files are
|
||||
left as-is so the link still points at the on-disk copy.
|
||||
"""
|
||||
new_map = dict(resource_map)
|
||||
|
||||
def replace(match: re.Match) -> str:
|
||||
label, rel_path = match.group(1), match.group(2)
|
||||
resource_id = new_map.get(rel_path)
|
||||
if resource_id is None:
|
||||
file_path = (note_dir / rel_path).resolve()
|
||||
if not file_path.is_file():
|
||||
logger.warning("[joplin] Media file missing, leaving link: %s", rel_path)
|
||||
return match.group(0)
|
||||
resource_id = client.create_resource(file_path)
|
||||
new_map[rel_path] = resource_id
|
||||
return f"{label}(:/{resource_id})"
|
||||
|
||||
return _MEDIA_LINK_RE.sub(replace, body), new_map
|
||||
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Notebook naming helper
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
|
||||
_PROVIDER_DISPLAY = {
|
||||
"chatgpt": "ChatGPT",
|
||||
"claude": "Claude",
|
||||
"chatgpt": "AI-ChatGPT",
|
||||
"claude": "AI-Claude",
|
||||
# Decision 2026-06-12: Claude Code coding projects nest under the same
|
||||
# AI-Claude parent, alongside Claude web projects.
|
||||
"claude-code": "AI-Claude",
|
||||
}
|
||||
|
||||
|
||||
def notebook_title(provider: str, project: str | None) -> str:
|
||||
"""Derive a flat Joplin notebook title from provider and project name.
|
||||
def notebook_path(provider: str, project: str | None) -> tuple[str, str]:
|
||||
"""Return (parent_notebook, child_notebook) for the given provider and project.
|
||||
|
||||
The parent is the top-level provider notebook; the child is the project name.
|
||||
|
||||
Examples:
|
||||
notebook_title("chatgpt", "no-project") → "ChatGPT - No Project"
|
||||
notebook_title("claude", "budget-tracker") → "Claude - Budget Tracker"
|
||||
notebook_title("chatgpt", None) → "ChatGPT - No Project"
|
||||
notebook_path("chatgpt", None) → ("AI-ChatGPT", "No Project")
|
||||
notebook_path("chatgpt", "no-project") → ("AI-ChatGPT", "No Project")
|
||||
notebook_path("claude", "budget-tracker") → ("AI-Claude", "Budget Tracker")
|
||||
"""
|
||||
prov_display = _PROVIDER_DISPLAY.get(provider, provider.capitalize())
|
||||
proj = (project or "no-project").replace("-", " ").title()
|
||||
return f"{prov_display} - {proj}"
|
||||
parent = _PROVIDER_DISPLAY.get(provider, f"AI-{provider.capitalize()}")
|
||||
child = (project or "no-project").replace("-", " ").title()
|
||||
return parent, child
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Per-export-run tally for content that was dropped or partially extracted.
|
||||
|
||||
Surfaces the loss visibility that the rest of the system promises in its
|
||||
output (visible ``unknown`` blocks). The summary emitted at the end of
|
||||
each export is the load-bearing operator-facing signal: if a real content
|
||||
type starts being silently dropped, this is where it shows up.
|
||||
|
||||
Pass a single instance through ``BaseProvider.normalize_conversation`` and
|
||||
read it back in ``src/main.py`` after the export loop. No global state.
|
||||
"""
|
||||
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
_TOP_N_BREAKDOWN = 5
|
||||
|
||||
|
||||
@dataclass
|
||||
class LossReport:
|
||||
"""Counters for things that didn't render cleanly in an export run."""
|
||||
|
||||
# Type-keyed counters. Values are int counts.
|
||||
unknown_blocks: Counter = field(default_factory=Counter)
|
||||
extraction_failures: Counter = field(default_factory=Counter)
|
||||
filtered_roles: Counter = field(default_factory=Counter)
|
||||
# Messages collapsed/omitted by the EXPORTER_HIDDEN_CONTENT policy —
|
||||
# intentional, but still surfaced so the omission is never invisible.
|
||||
collapsed: Counter = field(default_factory=Counter)
|
||||
collapsed_bytes: int = 0
|
||||
|
||||
# Aggregate counters
|
||||
messages_rendered: int = 0
|
||||
conversations: int = 0
|
||||
|
||||
# Media downloads (EXPORTER_DOWNLOAD_MEDIA): successes / skips / failures
|
||||
media_downloaded: int = 0
|
||||
media_downloaded_bytes: int = 0
|
||||
media_failed: Counter = field(default_factory=Counter)
|
||||
|
||||
# Recording -------------------------------------------------------------
|
||||
|
||||
def record_unknown(self, raw_type: str) -> None:
|
||||
self.unknown_blocks[raw_type or "?"] += 1
|
||||
|
||||
def record_extraction_failure(self, raw_type: str) -> None:
|
||||
self.extraction_failures[raw_type or "?"] += 1
|
||||
|
||||
def record_filtered_role(self, role: str) -> None:
|
||||
self.filtered_roles[role or "?"] += 1
|
||||
|
||||
def record_collapsed(self, origin: str, size_bytes: int = 0) -> None:
|
||||
self.collapsed[origin or "?"] += 1
|
||||
if isinstance(size_bytes, int) and size_bytes > 0:
|
||||
self.collapsed_bytes += size_bytes
|
||||
|
||||
def record_media_downloaded(self, size_bytes: int = 0) -> None:
|
||||
self.media_downloaded += 1
|
||||
if isinstance(size_bytes, int) and size_bytes > 0:
|
||||
self.media_downloaded_bytes += size_bytes
|
||||
|
||||
def record_media_failed(self, reason: str) -> None:
|
||||
self.media_failed[reason or "?"] += 1
|
||||
|
||||
def record_message(self) -> None:
|
||||
self.messages_rendered += 1
|
||||
|
||||
def record_conversation(self) -> None:
|
||||
self.conversations += 1
|
||||
|
||||
# Summary ---------------------------------------------------------------
|
||||
|
||||
def format_summary(self) -> str:
|
||||
"""Return a multi-line summary table suitable for INFO logging.
|
||||
|
||||
Format pinned by plan §Post-export summary — "(none)" sentinel when a
|
||||
counter is empty, top-5 breakdown with "+ N more types" overflow.
|
||||
"""
|
||||
lines: list[str] = ["[export] Run summary:"]
|
||||
lines.append(f" conversations: {self.conversations}")
|
||||
lines.append(f" messages rendered: {self.messages_rendered}")
|
||||
lines.extend(_format_section("unknown blocks: ", self.unknown_blocks))
|
||||
lines.extend(_format_section("extraction failures: ", self.extraction_failures))
|
||||
lines.extend(_format_section("collapsed by policy: ", self.collapsed))
|
||||
if self.collapsed:
|
||||
lines.append(
|
||||
f" ≈{self.collapsed_bytes / 1024:.0f} KB omitted "
|
||||
"(intentional — EXPORTER_HIDDEN_CONTENT=full to keep)"
|
||||
)
|
||||
if self.media_downloaded or self.media_failed:
|
||||
lines.append(
|
||||
f" media downloaded: {self.media_downloaded} "
|
||||
f"(≈{self.media_downloaded_bytes / 1024:.0f} KB)"
|
||||
)
|
||||
if self.media_failed:
|
||||
total_failed = sum(self.media_failed.values())
|
||||
lines.append(f" media failed: {total_failed}")
|
||||
for reason, count in self.media_failed.most_common(_TOP_N_BREAKDOWN):
|
||||
lines.append(f" {reason}={count}")
|
||||
lines.append(
|
||||
" filtered roles: "
|
||||
"(filter lifted in v0.4.0 — counter retained for future use, expected 0)"
|
||||
)
|
||||
if self.filtered_roles:
|
||||
for role, count in self.filtered_roles.most_common(_TOP_N_BREAKDOWN):
|
||||
lines.append(f" {role}={count}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _format_section(label: str, counter: Counter) -> list[str]:
|
||||
"""Render one counter section: header line + indented breakdown lines."""
|
||||
total = sum(counter.values())
|
||||
header = f" {label} {total}"
|
||||
if total == 0:
|
||||
return [header, " (none)"]
|
||||
|
||||
lines = [header]
|
||||
most_common = counter.most_common()
|
||||
for raw_type, count in most_common[:_TOP_N_BREAKDOWN]:
|
||||
lines.append(f" {raw_type}={count}")
|
||||
if len(most_common) > _TOP_N_BREAKDOWN:
|
||||
remainder = len(most_common) - _TOP_N_BREAKDOWN
|
||||
lines.append(f" + {remainder} more types")
|
||||
return lines
|
||||
+365
-29
@@ -2,6 +2,7 @@
|
||||
|
||||
import importlib.metadata
|
||||
import logging
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import sys
|
||||
@@ -16,6 +17,7 @@ from rich.table import Table
|
||||
from src.cache import Cache, CacheError
|
||||
from src.config import ConfigError
|
||||
from src.logging_config import setup_logging
|
||||
from src.loss_report import LossReport
|
||||
from src.providers.base import ProviderError
|
||||
|
||||
console = Console()
|
||||
@@ -108,17 +110,37 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, debug: bool, no_log_file
|
||||
|
||||
|
||||
@cli.command()
|
||||
@click.option(
|
||||
"--from-browser",
|
||||
"from_browser",
|
||||
is_flag=False,
|
||||
flag_value="brave",
|
||||
default=None,
|
||||
metavar="[BROWSER]",
|
||||
help=(
|
||||
"Extract tokens straight from a local browser's cookie store instead "
|
||||
"of the manual DevTools flow. Defaults to brave when no browser is "
|
||||
"named; also supports chrome, chromium, edge, firefox. The browser "
|
||||
"must be installed and logged in on this machine."
|
||||
),
|
||||
)
|
||||
@click.pass_context
|
||||
def auth(ctx: click.Context) -> None:
|
||||
def auth(ctx: click.Context, from_browser: str | None) -> None:
|
||||
"""Interactive setup wizard for session tokens.
|
||||
|
||||
Guides you through finding and saving your ChatGPT and Claude session
|
||||
tokens. Tokens are never echoed to the terminal.
|
||||
tokens. Tokens are never echoed to the terminal. With --from-browser,
|
||||
tokens are pulled from the browser's cookie store and written to .env
|
||||
without any copy-pasting.
|
||||
|
||||
Token lifetimes:
|
||||
ChatGPT (__Secure-next-auth.session-token): ~7 days (JWT)
|
||||
Claude (sessionKey): ~30 days (opaque string)
|
||||
"""
|
||||
if from_browser:
|
||||
_auth_from_browser(from_browser.lower())
|
||||
return
|
||||
|
||||
os_name = platform.system()
|
||||
|
||||
console.print("\n[bold cyan]AI Chat Exporter — Token Setup Wizard[/bold cyan]\n")
|
||||
@@ -278,11 +300,91 @@ def _auth_claude(os_name: str) -> None:
|
||||
_write_token_to_env("CLAUDE_SESSION_KEY", key)
|
||||
|
||||
|
||||
def _auth_from_browser(browser: str) -> None:
|
||||
"""Extract ChatGPT + Claude tokens from a local browser and write .env.
|
||||
|
||||
Each provider is validated against its live API before anything is
|
||||
written, so a stale cookie (logged out since last visit) never replaces
|
||||
a working token. Exits 1 only when nothing could be configured.
|
||||
"""
|
||||
from src.browser_tokens import (
|
||||
BrowserTokenError,
|
||||
SUPPORTED_BROWSERS,
|
||||
extract_chatgpt_tokens,
|
||||
extract_claude_session,
|
||||
)
|
||||
|
||||
if browser not in SUPPORTED_BROWSERS:
|
||||
err_console.print(
|
||||
f"[red]Unsupported browser '{browser}'. "
|
||||
f"Supported: {', '.join(SUPPORTED_BROWSERS)}[/red]"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
console.print(f"\n[bold cyan]Extracting session tokens from {browser}…[/bold cyan]\n")
|
||||
configured = 0
|
||||
|
||||
# ── ChatGPT ──────────────────────────────────────────────────────────
|
||||
try:
|
||||
token, token_1 = extract_chatgpt_tokens(browser)
|
||||
with console.status("[dim]Validating ChatGPT token…[/dim]"):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
prov = ChatGPTProvider(session_token=token, session_token_1=token_1)
|
||||
prov._fetch_access_token()
|
||||
_set_env_key("CHATGPT_SESSION_TOKEN", token)
|
||||
_set_env_key("CHATGPT_SESSION_TOKEN_1", token_1 or "")
|
||||
console.print("[green]ChatGPT: token extracted, validated, and saved.[/green]")
|
||||
configured += 1
|
||||
except BrowserTokenError as e:
|
||||
console.print(f"[yellow]ChatGPT: {e}[/yellow]")
|
||||
except ProviderError as e:
|
||||
console.print(
|
||||
f"[yellow]ChatGPT: extracted cookie failed live validation "
|
||||
f"({e.original}) — .env not changed. Log in to chatgpt.com in "
|
||||
f"{browser} and retry.[/yellow]"
|
||||
)
|
||||
|
||||
# ── Claude ───────────────────────────────────────────────────────────
|
||||
try:
|
||||
session_key = extract_claude_session(browser)
|
||||
with console.status("[dim]Validating Claude session key…[/dim]"):
|
||||
from src.providers.claude import ClaudeProvider
|
||||
prov = ClaudeProvider(session_key)
|
||||
prov.list_conversations(offset=0, limit=1)
|
||||
_set_env_key("CLAUDE_SESSION_KEY", session_key)
|
||||
console.print("[green]Claude: session key extracted, validated, and saved.[/green]")
|
||||
configured += 1
|
||||
except BrowserTokenError as e:
|
||||
console.print(f"[yellow]Claude: {e}[/yellow]")
|
||||
except ProviderError as e:
|
||||
console.print(
|
||||
f"[yellow]Claude: extracted cookie failed live validation "
|
||||
f"({e.original}) — .env not changed. Log in to claude.ai in "
|
||||
f"{browser} and retry.[/yellow]"
|
||||
)
|
||||
|
||||
if configured:
|
||||
console.print(
|
||||
f"\n[green]Done — {configured} provider(s) configured. "
|
||||
"Run 'ai-chat-exporter doctor' to verify.[/green]"
|
||||
)
|
||||
else:
|
||||
err_console.print(
|
||||
"\n[red]No tokens could be extracted. Use the manual wizard "
|
||||
"instead: ai-chat-exporter auth[/red]"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def _write_token_to_env(key: str, value: str) -> None:
|
||||
"""Write or update a key in .env, offering to create the file if it doesn't exist."""
|
||||
env_path = Path(".env")
|
||||
|
||||
if click.confirm(f"Write {key} to .env?", default=True):
|
||||
_set_env_key(key, value)
|
||||
|
||||
|
||||
def _set_env_key(key: str, value: str) -> None:
|
||||
"""Write or update a key in .env without prompting (creates .env if needed)."""
|
||||
env_path = Path(".env")
|
||||
if not env_path.exists():
|
||||
# Create from example if available
|
||||
example = Path(".env.example")
|
||||
@@ -307,7 +409,6 @@ def _write_token_to_env(key: str, value: str) -> None:
|
||||
new_lines.append(f"\n{key}={value}\n")
|
||||
|
||||
env_path.write_text("".join(new_lines), encoding="utf-8")
|
||||
import os
|
||||
os.chmod(env_path, 0o600)
|
||||
console.print(f"[green]{key} written to .env (permissions: 600)[/green]")
|
||||
|
||||
@@ -325,15 +426,19 @@ def doctor(ctx: click.Context) -> None:
|
||||
Checks token presence, format, expiry, directory permissions, disk space,
|
||||
and live API reachability. Exits with code 1 if any checks fail.
|
||||
"""
|
||||
checks = _run_doctor_checks()
|
||||
checks = _run_doctor_checks(cache=ctx.obj["cache"])
|
||||
_print_doctor_table(checks)
|
||||
|
||||
if any(not c["pass"] for c in checks):
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def _run_doctor_checks() -> list[dict]:
|
||||
"""Run all doctor checks and return results."""
|
||||
def _run_doctor_checks(cache: Cache | None = None) -> list[dict]:
|
||||
"""Run all doctor checks and return results.
|
||||
|
||||
``cache``: when provided, also verify manifest ↔ disk integrity (every
|
||||
``file_path`` recorded in the manifest exists on disk).
|
||||
"""
|
||||
import os
|
||||
import jwt as pyjwt
|
||||
from datetime import timezone
|
||||
@@ -404,6 +509,18 @@ def _run_doctor_checks() -> list[dict]:
|
||||
except OSError as e:
|
||||
add("Disk space check", False, str(e))
|
||||
|
||||
# Manifest ↔ disk integrity
|
||||
if cache is not None:
|
||||
referenced = cache.all_file_paths()
|
||||
if referenced:
|
||||
missing = sorted(
|
||||
p for p in referenced if not Path(p).expanduser().exists()
|
||||
)
|
||||
detail = f"{len(referenced) - len(missing)}/{len(referenced)} manifest files present"
|
||||
if missing:
|
||||
detail += f"; first missing: {missing[0]} — re-export to fix"
|
||||
add("Manifest files on disk", not missing, detail)
|
||||
|
||||
# API reachability
|
||||
if chatgpt_token:
|
||||
try:
|
||||
@@ -452,7 +569,7 @@ def _print_doctor_table(checks: list[dict]) -> None:
|
||||
@cli.command()
|
||||
@click.option(
|
||||
"--provider",
|
||||
type=click.Choice(["chatgpt", "claude", "all"], case_sensitive=False),
|
||||
type=click.Choice(["chatgpt", "claude", "claude-code", "all"], case_sensitive=False),
|
||||
default="all",
|
||||
show_default=True,
|
||||
help="Which provider to export.",
|
||||
@@ -486,6 +603,37 @@ def _print_doctor_table(checks: list[dict]) -> None:
|
||||
"Use 'none' for conversations outside any project."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--hidden-content",
|
||||
type=click.Choice(["full", "placeholder", "omit"], case_sensitive=False),
|
||||
default=None,
|
||||
help=(
|
||||
"What to do with content that was invisible in the provider's web UI "
|
||||
"(file-retrieval tool dumps, hidden context): keep in full, collapse to "
|
||||
"a one-line placeholder, or omit. Overrides EXPORTER_HIDDEN_CONTENT "
|
||||
"(default: placeholder)."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--max-conversations",
|
||||
type=click.IntRange(min=1),
|
||||
default=None,
|
||||
help=(
|
||||
"Cap how many conversations are downloaded this run (per provider). "
|
||||
"Runs are resumable, so re-running continues where the cap stopped. "
|
||||
"Overrides MAX_CONVERSATIONS_PER_RUN (default: unlimited)."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--download-media",
|
||||
type=click.Choice(["images", "all", "off"], case_sensitive=False),
|
||||
default=None,
|
||||
help=(
|
||||
"Download conversation assets next to the exports: images only, "
|
||||
"all (also audio/files), or off. Overrides EXPORTER_DOWNLOAD_MEDIA "
|
||||
"(default: images)."
|
||||
),
|
||||
)
|
||||
@click.option("--dry-run", is_flag=True, help="Show what would be exported without writing anything.")
|
||||
@click.pass_context
|
||||
def export(
|
||||
@@ -495,6 +643,9 @@ def export(
|
||||
output_dir: str | None,
|
||||
since: str | None,
|
||||
project_filter: str | None,
|
||||
hidden_content: str | None,
|
||||
max_conversations: int | None,
|
||||
download_media: str | None,
|
||||
dry_run: bool,
|
||||
) -> None:
|
||||
"""Export new and updated conversations to Markdown or JSON.
|
||||
@@ -506,6 +657,13 @@ def export(
|
||||
debug = ctx.obj.get("debug", False)
|
||||
cache: Cache = ctx.obj["cache"]
|
||||
|
||||
# CLI flags win over .env: set before load_config (load_dotenv uses
|
||||
# override=False, so an existing env var is not clobbered by .env).
|
||||
if hidden_content:
|
||||
os.environ["EXPORTER_HIDDEN_CONTENT"] = hidden_content.lower()
|
||||
if download_media:
|
||||
os.environ["EXPORTER_DOWNLOAD_MEDIA"] = download_media.lower()
|
||||
|
||||
# Load config (may raise ConfigError)
|
||||
try:
|
||||
from src.config import load_config
|
||||
@@ -516,7 +674,7 @@ def export(
|
||||
# First-run: auto-doctor
|
||||
if not cache.last_run():
|
||||
console.print("[dim]First run — checking configuration…[/dim]")
|
||||
checks = _run_doctor_checks()
|
||||
checks = _run_doctor_checks(cache=cache)
|
||||
_print_doctor_table(checks)
|
||||
if any(not c["pass"] for c in checks):
|
||||
err_console.print(
|
||||
@@ -536,6 +694,9 @@ def export(
|
||||
err_console.print(f"[red]Invalid --since date: '{since}'. Use YYYY-MM-DD.[/red]")
|
||||
sys.exit(1)
|
||||
|
||||
# Session cap: CLI flag wins over MAX_CONVERSATIONS_PER_RUN env default.
|
||||
session_cap = max_conversations if max_conversations is not None else cfg.max_conversations
|
||||
|
||||
# Determine which providers to run
|
||||
providers_to_run = _resolve_providers(provider, cfg)
|
||||
if not providers_to_run:
|
||||
@@ -554,6 +715,9 @@ def export(
|
||||
# Summary counters
|
||||
summary: dict[str, dict[str, int]] = {}
|
||||
|
||||
# Single LossReport tracks data-loss visibility across all providers in this run.
|
||||
loss_report = LossReport()
|
||||
|
||||
for prov_name, prov_instance in providers_to_run:
|
||||
summary[prov_name] = {"exported": 0, "skipped": 0, "failed": 0}
|
||||
|
||||
@@ -576,14 +740,33 @@ def export(
|
||||
skipped = len(all_convs) - len(to_export)
|
||||
summary[prov_name]["skipped"] = skipped
|
||||
|
||||
# Apply the session cap. Resumability makes this safe: the manifest
|
||||
# records each conversation immediately, so the next run picks up
|
||||
# exactly where the cap stopped.
|
||||
deferred = 0
|
||||
if session_cap is not None and len(to_export) > session_cap:
|
||||
deferred = len(to_export) - session_cap
|
||||
to_export = to_export[:session_cap]
|
||||
summary[prov_name]["deferred"] = deferred
|
||||
|
||||
if dry_run:
|
||||
_print_dry_run_table(prov_name, to_export, prov_instance, export_base, structure, skipped)
|
||||
if deferred:
|
||||
console.print(
|
||||
f" [yellow]Session cap {session_cap}: {deferred} more pending beyond this run.[/yellow]"
|
||||
)
|
||||
continue
|
||||
|
||||
if not to_export:
|
||||
console.print(f" [dim]{skipped} conversations already up to date.[/dim]")
|
||||
continue
|
||||
|
||||
if deferred:
|
||||
console.print(
|
||||
f" [dim]{len(to_export)} to export (session cap; {deferred} deferred), "
|
||||
f"{skipped} already up to date.[/dim]"
|
||||
)
|
||||
else:
|
||||
console.print(f" [dim]{len(to_export)} to export, {skipped} already up to date.[/dim]")
|
||||
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, TaskProgressColumn
|
||||
@@ -611,7 +794,20 @@ def export(
|
||||
for key, val in raw_conv.items():
|
||||
if (key.startswith("_") or key in _PROPAGATE_KEYS) and key not in full_raw:
|
||||
full_raw[key] = val
|
||||
normalized = prov_instance.normalize_conversation(full_raw)
|
||||
normalized = prov_instance.normalize_conversation(full_raw, loss_report)
|
||||
|
||||
# Download assets (images by default) so the renderers
|
||||
# can inline them. Failures keep placeholders; never fatal.
|
||||
if cfg.download_media != "off":
|
||||
from src.media import resolve_media
|
||||
resolve_media(
|
||||
normalized,
|
||||
prov_instance,
|
||||
export_base,
|
||||
structure,
|
||||
cfg.download_media,
|
||||
loss_report,
|
||||
)
|
||||
|
||||
exported_path: Path | None = None
|
||||
if md_exporter:
|
||||
@@ -623,6 +819,7 @@ def export(
|
||||
cache.mark_exported(prov_name, conv_id, {
|
||||
"title": normalized.get("title", ""),
|
||||
"project": normalized.get("project"),
|
||||
"created_at": normalized.get("created_at", ""),
|
||||
"updated_at": normalized.get("updated_at", ""),
|
||||
"file_path": str(exported_path) if exported_path else "",
|
||||
})
|
||||
@@ -642,6 +839,16 @@ def export(
|
||||
|
||||
if not dry_run:
|
||||
_print_export_summary(summary)
|
||||
total_deferred = sum(s.get("deferred", 0) for s in summary.values())
|
||||
if total_deferred:
|
||||
console.print(
|
||||
f"[yellow]{total_deferred} conversation(s) deferred by the session cap "
|
||||
f"({session_cap} per provider per run). Re-run the same command to continue.[/yellow]"
|
||||
)
|
||||
# Emit the data-loss summary at INFO level so it lands in the log file
|
||||
# AND the operator's console (default level is INFO).
|
||||
for line in loss_report.format_summary().split("\n"):
|
||||
logger.info(line)
|
||||
|
||||
|
||||
def _resolve_providers(provider: str, cfg) -> list[tuple[str, object]]:
|
||||
@@ -674,6 +881,7 @@ def _resolve_providers(provider: str, cfg) -> list[tuple[str, object]]:
|
||||
session_token=cfg.chatgpt_session_token,
|
||||
session_token_1=cfg.chatgpt_session_token_1,
|
||||
project_ids=cfg.chatgpt_project_ids,
|
||||
hidden_content=cfg.hidden_content,
|
||||
),
|
||||
))
|
||||
except ProviderError as e:
|
||||
@@ -685,6 +893,19 @@ def _resolve_providers(provider: str, cfg) -> list[tuple[str, object]]:
|
||||
if provider in ("claude", "all"):
|
||||
try_add("claude", cfg.claude_session_key, ClaudeProvider)
|
||||
|
||||
if provider in ("claude-code", "all"):
|
||||
from src.providers.claude_code import ClaudeCodeProvider, DEFAULT_PROJECTS_DIR
|
||||
cc_dir = Path(os.getenv("CLAUDE_CODE_DIR", DEFAULT_PROJECTS_DIR)).expanduser()
|
||||
if cc_dir.is_dir():
|
||||
result.append((
|
||||
"claude-code",
|
||||
ClaudeCodeProvider(projects_dir=cc_dir, hidden_content=cfg.hidden_content),
|
||||
))
|
||||
elif provider == "claude-code":
|
||||
logging.getLogger(__name__).warning(
|
||||
"[claude-code] Skipping — %s not found (set CLAUDE_CODE_DIR).", cc_dir
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@@ -781,7 +1002,7 @@ def _print_export_summary(summary: dict[str, dict[str, int]]) -> None:
|
||||
@cli.command(name="list")
|
||||
@click.option(
|
||||
"--provider",
|
||||
type=click.Choice(["chatgpt", "claude", "all"], case_sensitive=False),
|
||||
type=click.Choice(["chatgpt", "claude", "claude-code", "all"], case_sensitive=False),
|
||||
default="all",
|
||||
show_default=True,
|
||||
)
|
||||
@@ -847,7 +1068,7 @@ def list_conversations(ctx: click.Context, provider: str, project_filter: str |
|
||||
@click.option("--clear", is_flag=True, help="Clear cached entries.")
|
||||
@click.option(
|
||||
"--provider",
|
||||
type=click.Choice(["chatgpt", "claude", "all"], case_sensitive=False),
|
||||
type=click.Choice(["chatgpt", "claude", "claude-code", "all"], case_sensitive=False),
|
||||
default="all",
|
||||
help="Provider to target (used with --clear).",
|
||||
)
|
||||
@@ -877,6 +1098,106 @@ def cache(ctx: click.Context, show: bool, clear: bool, provider: str) -> None:
|
||||
console.print("Specify --show or --clear. Use --help for options.")
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# prune command
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@cli.command()
|
||||
@click.option("--dry-run", is_flag=True, help="List stale files without deleting anything.")
|
||||
@click.option("--yes", "-y", is_flag=True, help="Delete without a confirmation prompt.")
|
||||
@click.pass_context
|
||||
def prune(ctx: click.Context, dry_run: bool, yes: bool) -> None:
|
||||
"""Delete export files no longer referenced by the cache manifest.
|
||||
|
||||
Removes leftovers from old folder layouts and orphaned files from fixed
|
||||
bugs (e.g. pre-v0.5.0 nested year folders, files with a missing
|
||||
conversation ID). Only .md and .json files under EXPORT_DIR are
|
||||
considered; the manifest is the source of truth.
|
||||
"""
|
||||
cache_obj: Cache = ctx.obj["cache"]
|
||||
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv(override=False)
|
||||
export_dir = Path(os.getenv("EXPORT_DIR", "./exports")).expanduser()
|
||||
|
||||
if not export_dir.exists():
|
||||
console.print(f"[yellow]Export directory {export_dir} does not exist — nothing to prune.[/yellow]")
|
||||
return
|
||||
|
||||
referenced_raw = cache_obj.all_file_paths()
|
||||
if not referenced_raw:
|
||||
# Footgun guard: with an empty manifest (e.g. right after `cache
|
||||
# --clear`) every export file would look stale and prune would wipe
|
||||
# the entire archive.
|
||||
err_console.print(
|
||||
"[red]The manifest references no export files (did you just run "
|
||||
"'cache --clear'?). Refusing to prune — run an export first.[/red]"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
referenced = {Path(p).expanduser().resolve() for p in referenced_raw}
|
||||
on_disk = sorted(
|
||||
f for f in export_dir.rglob("*")
|
||||
if f.is_file() and f.suffix in (".md", ".json")
|
||||
)
|
||||
stale = [f for f in on_disk if f.resolve() not in referenced]
|
||||
|
||||
if not stale:
|
||||
console.print(
|
||||
f"[green]Nothing to prune — all {len(on_disk)} export files are referenced "
|
||||
"by the manifest.[/green]"
|
||||
)
|
||||
return
|
||||
|
||||
total_bytes = sum(f.stat().st_size for f in stale)
|
||||
console.print(
|
||||
f"[bold]{len(stale)} stale file(s)[/bold] "
|
||||
f"({total_bytes / (1024 * 1024):.1f} MB) not referenced by the manifest "
|
||||
f"(of {len(on_disk)} total):\n"
|
||||
)
|
||||
for f in stale:
|
||||
try:
|
||||
shown = f.relative_to(export_dir.resolve()) if f.is_absolute() else f
|
||||
except ValueError:
|
||||
shown = f
|
||||
console.print(f" [dim]{shown}[/dim]")
|
||||
|
||||
if dry_run:
|
||||
console.print("\n[yellow]Dry run — nothing deleted.[/yellow]")
|
||||
return
|
||||
|
||||
if not yes and not click.confirm(f"\nDelete these {len(stale)} files?", default=False):
|
||||
console.print("Aborted — nothing deleted.")
|
||||
return
|
||||
|
||||
deleted = 0
|
||||
for f in stale:
|
||||
try:
|
||||
f.unlink()
|
||||
deleted += 1
|
||||
except OSError as e:
|
||||
logger.error("[prune] Could not delete %s: %s", f, e)
|
||||
|
||||
# Sweep now-empty directories (deepest first).
|
||||
removed_dirs = 0
|
||||
for d in sorted(
|
||||
(d for d in export_dir.rglob("*") if d.is_dir()),
|
||||
key=lambda p: len(p.parts),
|
||||
reverse=True,
|
||||
):
|
||||
try:
|
||||
d.rmdir() # only succeeds when empty
|
||||
removed_dirs += 1
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
console.print(
|
||||
f"[green]Deleted {deleted} file(s) ({total_bytes / (1024 * 1024):.1f} MB) "
|
||||
f"and {removed_dirs} empty director(ies).[/green]"
|
||||
)
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# joplin command
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
@@ -885,7 +1206,7 @@ def cache(ctx: click.Context, show: bool, clear: bool, provider: str) -> None:
|
||||
@cli.command()
|
||||
@click.option(
|
||||
"--provider",
|
||||
type=click.Choice(["chatgpt", "claude", "all"], case_sensitive=False),
|
||||
type=click.Choice(["chatgpt", "claude", "claude-code", "all"], case_sensitive=False),
|
||||
default="all",
|
||||
show_default=True,
|
||||
help="Which provider's conversations to sync to Joplin.",
|
||||
@@ -908,9 +1229,9 @@ def joplin(ctx: click.Context, provider: str, project_filter: str | None, dry_ru
|
||||
via its local REST API. Requires Joplin desktop to be running with the
|
||||
Web Clipper service enabled.
|
||||
|
||||
Notebooks are created automatically based on provider and project:
|
||||
exports/chatgpt/my-project/ → "ChatGPT - My Project" notebook
|
||||
exports/claude/no-project/ → "Claude - No Project" notebook
|
||||
Notebooks are created automatically as nested folders:
|
||||
chatgpt / my-project → AI-ChatGPT / My Project
|
||||
claude / no-project → AI-Claude / No Project
|
||||
|
||||
Re-running is safe: notes are updated (not duplicated) on subsequent runs.
|
||||
|
||||
@@ -936,7 +1257,7 @@ def joplin(ctx: click.Context, provider: str, project_filter: str | None, dry_ru
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
from src.joplin import JoplinClient, JoplinError, notebook_title
|
||||
from src.joplin import JoplinClient, JoplinError, notebook_path
|
||||
|
||||
client = JoplinClient(cfg.joplin_api_url, cfg.joplin_api_token)
|
||||
|
||||
@@ -958,10 +1279,9 @@ def joplin(ctx: click.Context, provider: str, project_filter: str | None, dry_ru
|
||||
|
||||
# Determine which providers to process
|
||||
providers_to_sync: list[str] = []
|
||||
if provider in ("chatgpt", "all"):
|
||||
providers_to_sync.append("chatgpt")
|
||||
if provider in ("claude", "all"):
|
||||
providers_to_sync.append("claude")
|
||||
for prov in ("chatgpt", "claude", "claude-code"):
|
||||
if provider in (prov, "all"):
|
||||
providers_to_sync.append(prov)
|
||||
|
||||
summary: dict[str, dict[str, int]] = {}
|
||||
|
||||
@@ -1016,7 +1336,9 @@ def joplin(ctx: click.Context, provider: str, project_filter: str | None, dry_ru
|
||||
|
||||
for conv_id, entry in pending:
|
||||
file_path = entry.get("file_path", "")
|
||||
title = entry.get("title") or "Untitled"
|
||||
raw_title = entry.get("title") or "Untitled"
|
||||
created_date = (entry.get("created_at") or "")[:10]
|
||||
title = f"{created_date} {raw_title}" if created_date else raw_title
|
||||
project = entry.get("project") or None
|
||||
existing_note_id = entry.get("joplin_note_id")
|
||||
action = "update" if existing_note_id else "create"
|
||||
@@ -1031,9 +1353,20 @@ def joplin(ctx: click.Context, provider: str, project_filter: str | None, dry_ru
|
||||
body = Path(file_path).read_text(encoding="utf-8")
|
||||
logger.debug("[joplin] Read %d chars from %s", len(body), file_path)
|
||||
|
||||
# Get or create the notebook
|
||||
nb_title = notebook_title(prov_name, project)
|
||||
notebook_id = client.get_or_create_notebook(nb_title)
|
||||
# Upload any downloaded media as Joplin resources and
|
||||
# rewrite local media/ links to :/resourceId references.
|
||||
# The cache stores the resource map so re-syncs reuse IDs.
|
||||
from src.joplin import upload_media_and_rewrite
|
||||
res_map = cache_obj.get_joplin_resources(prov_name, conv_id)
|
||||
body, new_res_map = upload_media_and_rewrite(
|
||||
body, Path(file_path).parent, client, res_map
|
||||
)
|
||||
if new_res_map != res_map:
|
||||
cache_obj.set_joplin_resources(prov_name, conv_id, new_res_map)
|
||||
|
||||
# Get or create the nested notebook
|
||||
nb_path = notebook_path(prov_name, project)
|
||||
notebook_id = client.get_or_create_notebook_path(list(nb_path))
|
||||
|
||||
if existing_note_id:
|
||||
client.update_note(existing_note_id, title, body)
|
||||
@@ -1070,7 +1403,7 @@ def joplin(ctx: click.Context, provider: str, project_filter: str | None, dry_ru
|
||||
|
||||
|
||||
def _print_joplin_dry_run_table(prov_name: str, pending: list[tuple[str, dict]]) -> None:
|
||||
from src.joplin import notebook_title
|
||||
from src.joplin import notebook_path
|
||||
|
||||
table = Table(title=f"[DRY RUN] {prov_name.upper()} — Would sync {len(pending)} conversation(s)")
|
||||
table.add_column("Title")
|
||||
@@ -1079,9 +1412,12 @@ def _print_joplin_dry_run_table(prov_name: str, pending: list[tuple[str, dict]])
|
||||
table.add_column("Action")
|
||||
|
||||
for conv_id, entry in pending[:50]:
|
||||
title = entry.get("title") or "Untitled"
|
||||
raw_title = entry.get("title") or "Untitled"
|
||||
created_date = (entry.get("created_at") or "")[:10]
|
||||
title = f"{created_date} {raw_title}" if created_date else raw_title
|
||||
project = entry.get("project") or "no-project"
|
||||
nb = notebook_title(prov_name, entry.get("project"))
|
||||
parent, child = notebook_path(prov_name, entry.get("project"))
|
||||
nb = f"{parent} / {child}"
|
||||
action = "update" if entry.get("joplin_note_id") else "create"
|
||||
table.add_row(title[:50], project[:30], nb, action)
|
||||
|
||||
|
||||
+202
@@ -0,0 +1,202 @@
|
||||
"""Media resolution — download conversation assets next to the export files.
|
||||
|
||||
Walks a normalized conversation's placeholder blocks, downloads each asset via
|
||||
the provider, saves it under a ``media/`` directory beside the conversation's
|
||||
Markdown file, and annotates the block with a relative ``local_path`` so the
|
||||
renderer emits a real inline image / file link instead of a placeholder.
|
||||
|
||||
Policy (``EXPORTER_DOWNLOAD_MEDIA``):
|
||||
images (default) — image_placeholder blocks only
|
||||
all — also file_placeholder blocks (voice-mode audio etc.;
|
||||
measured 2026-06-12: 556 clips ≈ 162MB whose transcripts
|
||||
are already in the exports as text)
|
||||
off — leave placeholders untouched
|
||||
|
||||
Failures (expired assets, network) keep the existing placeholder and are
|
||||
counted in the run summary — never fatal to the export.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from src.blocks import BLOCK_TYPE_FILE_PLACEHOLDER, BLOCK_TYPE_IMAGE_PLACEHOLDER
|
||||
from src.loss_report import LossReport
|
||||
from src.providers.base import ProviderError
|
||||
from src.utils import build_export_path, generate_filename
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MEDIA_IMAGES = "images"
|
||||
MEDIA_ALL = "all"
|
||||
MEDIA_OFF = "off"
|
||||
VALID_MEDIA_POLICIES = {MEDIA_IMAGES, MEDIA_ALL, MEDIA_OFF}
|
||||
|
||||
_EXT_BY_MIME = {
|
||||
"image/png": ".png",
|
||||
"image/jpeg": ".jpg",
|
||||
"image/gif": ".gif",
|
||||
"image/webp": ".webp",
|
||||
"audio/wav": ".wav",
|
||||
"audio/x-wav": ".wav",
|
||||
"audio/mpeg": ".mp3",
|
||||
"audio/mp4": ".m4a",
|
||||
"audio/aac": ".aac",
|
||||
"application/pdf": ".pdf",
|
||||
}
|
||||
|
||||
|
||||
def resolve_media_policy() -> str:
|
||||
"""Read EXPORTER_DOWNLOAD_MEDIA from the environment, defaulting to images."""
|
||||
value = os.getenv("EXPORTER_DOWNLOAD_MEDIA", "").strip().lower()
|
||||
if not value:
|
||||
return MEDIA_IMAGES
|
||||
if value not in VALID_MEDIA_POLICIES:
|
||||
logger.warning(
|
||||
"EXPORTER_DOWNLOAD_MEDIA=%r is invalid (expected images|all|off) "
|
||||
"— using 'images'.",
|
||||
value,
|
||||
)
|
||||
return MEDIA_IMAGES
|
||||
return value
|
||||
|
||||
|
||||
def resolve_media(
|
||||
normalized: dict,
|
||||
provider,
|
||||
export_base: Path,
|
||||
structure: str,
|
||||
policy: str,
|
||||
report: LossReport,
|
||||
) -> int:
|
||||
"""Download assets for one normalized conversation. Returns download count.
|
||||
|
||||
Mutates placeholder blocks in place (adds ``local_path``). Idempotent:
|
||||
an asset already on disk is annotated without an API call.
|
||||
"""
|
||||
if policy == MEDIA_OFF:
|
||||
return 0
|
||||
|
||||
download = getattr(provider, "download_asset", None)
|
||||
if download is None:
|
||||
# Provider has no remote assets (e.g. claude-code local transcripts).
|
||||
return 0
|
||||
|
||||
wanted_types = {BLOCK_TYPE_IMAGE_PLACEHOLDER}
|
||||
if policy == MEDIA_ALL:
|
||||
wanted_types.add(BLOCK_TYPE_FILE_PLACEHOLDER)
|
||||
|
||||
targets = [
|
||||
block
|
||||
for message in normalized.get("messages", [])
|
||||
for block in message.get("blocks", [])
|
||||
if block.get("type") in wanted_types and block.get("ref")
|
||||
]
|
||||
if not targets:
|
||||
return 0
|
||||
|
||||
# Same path computation the exporter uses, so media/ lands beside the .md.
|
||||
filename = generate_filename(
|
||||
normalized.get("title", "Untitled"),
|
||||
normalized.get("id", ""),
|
||||
normalized.get("created_at") or "2000-01-01",
|
||||
)
|
||||
conv_dir = build_export_path(
|
||||
export_base,
|
||||
normalized.get("provider", ""),
|
||||
normalized.get("project"),
|
||||
normalized.get("created_at") or "2000-01-01",
|
||||
filename,
|
||||
structure,
|
||||
).parent
|
||||
media_dir = conv_dir / "media"
|
||||
|
||||
downloaded = 0
|
||||
for block in targets:
|
||||
ref = block["ref"]
|
||||
file_id = _safe_asset_name(provider, ref)
|
||||
if not file_id:
|
||||
report.record_media_failed("unparseable-ref")
|
||||
continue
|
||||
|
||||
existing = _find_existing(media_dir, file_id)
|
||||
if existing is not None:
|
||||
block["local_path"] = f"media/{existing.name}"
|
||||
continue
|
||||
|
||||
try:
|
||||
content, mime, file_name = download(ref)
|
||||
except ProviderError as e:
|
||||
logger.warning(
|
||||
"[media] Could not download %s: %s", ref[:60], e.original
|
||||
)
|
||||
reason = "expired-or-missing" if "404" in str(e.original) or "not found" in str(
|
||||
e.original
|
||||
).lower() else "download-error"
|
||||
report.record_media_failed(reason)
|
||||
continue
|
||||
|
||||
ext = _pick_extension(mime, file_name)
|
||||
target = media_dir / f"{file_id}{ext}"
|
||||
try:
|
||||
_write_atomic(target, content)
|
||||
except OSError as e:
|
||||
logger.error("[media] Could not write %s: %s", target, e)
|
||||
report.record_media_failed("write-error")
|
||||
continue
|
||||
|
||||
block["local_path"] = f"media/{target.name}"
|
||||
report.record_media_downloaded(len(content))
|
||||
downloaded += 1
|
||||
|
||||
return downloaded
|
||||
|
||||
|
||||
def _safe_asset_name(provider, ref: str) -> str | None:
|
||||
"""A stable, filesystem-safe name for the asset (the provider file ID)."""
|
||||
parser = getattr(provider, "parse_asset_file_id", None)
|
||||
if parser is None:
|
||||
from src.providers.chatgpt import parse_asset_file_id as parser
|
||||
file_id = parser(ref)
|
||||
if not file_id:
|
||||
return None
|
||||
return "".join(c if c.isalnum() or c in "-_." else "_" for c in file_id)
|
||||
|
||||
|
||||
def _find_existing(media_dir: Path, file_id: str) -> Path | None:
|
||||
"""Return an already-downloaded file for this asset, if present."""
|
||||
if not media_dir.is_dir():
|
||||
return None
|
||||
for candidate in media_dir.glob(f"{file_id}.*"):
|
||||
if candidate.is_file() and candidate.stat().st_size > 0:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _pick_extension(mime: str | None, file_name: str | None) -> str:
|
||||
if mime:
|
||||
base_mime = mime.split(";")[0].strip().lower()
|
||||
if base_mime in _EXT_BY_MIME:
|
||||
return _EXT_BY_MIME[base_mime]
|
||||
if file_name:
|
||||
suffix = Path(file_name).suffix
|
||||
if suffix and len(suffix) <= 8:
|
||||
return suffix.lower()
|
||||
return ".bin"
|
||||
|
||||
|
||||
def _write_atomic(target: Path, content: bytes) -> None:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp_name = tempfile.mkstemp(dir=target.parent, suffix=".tmp")
|
||||
try:
|
||||
with os.fdopen(fd, "wb") as fh:
|
||||
fh.write(content)
|
||||
os.chmod(tmp_name, 0o600)
|
||||
os.replace(tmp_name, target)
|
||||
except OSError:
|
||||
try:
|
||||
os.unlink(tmp_name)
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
+87
-2
@@ -1,6 +1,7 @@
|
||||
"""Abstract base class for AI chat providers."""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import time
|
||||
from abc import ABC, abstractmethod
|
||||
@@ -9,6 +10,7 @@ from typing import Any
|
||||
|
||||
import requests
|
||||
|
||||
from src.loss_report import LossReport
|
||||
from src.utils import redact_secrets
|
||||
|
||||
# curl_cffi has its own exception hierarchy (rooted at CurlError → OSError),
|
||||
@@ -36,6 +38,62 @@ MAX_RETRIES = 3
|
||||
BACKOFF_BASE = 2.0
|
||||
BACKOFF_MAX = 60.0
|
||||
|
||||
# EXPORTER_HIDDEN_CONTENT policy — what to do with content that was invisible
|
||||
# in the provider's UI (retrieval dumps, hidden-flagged context, agent tool
|
||||
# traffic). Shared by all providers.
|
||||
HIDDEN_CONTENT_FULL = "full"
|
||||
HIDDEN_CONTENT_PLACEHOLDER = "placeholder"
|
||||
HIDDEN_CONTENT_OMIT = "omit"
|
||||
VALID_HIDDEN_CONTENT_POLICIES = {
|
||||
HIDDEN_CONTENT_FULL,
|
||||
HIDDEN_CONTENT_PLACEHOLDER,
|
||||
HIDDEN_CONTENT_OMIT,
|
||||
}
|
||||
|
||||
|
||||
def resolve_hidden_content_policy() -> str:
|
||||
"""Read EXPORTER_HIDDEN_CONTENT from the environment, defaulting to placeholder."""
|
||||
value = os.getenv("EXPORTER_HIDDEN_CONTENT", "").strip().lower()
|
||||
if not value:
|
||||
return HIDDEN_CONTENT_PLACEHOLDER
|
||||
if value not in VALID_HIDDEN_CONTENT_POLICIES:
|
||||
logger.warning(
|
||||
"EXPORTER_HIDDEN_CONTENT=%r is invalid (expected full|placeholder|omit) "
|
||||
"— using 'placeholder'.",
|
||||
value,
|
||||
)
|
||||
return HIDDEN_CONTENT_PLACEHOLDER
|
||||
return value
|
||||
|
||||
|
||||
# Default polite pacing between consecutive API requests (seconds). Keeps
|
||||
# traffic human-paced instead of bursty. REQUEST_DELAY=0 disables.
|
||||
DEFAULT_REQUEST_DELAY = 1.0
|
||||
|
||||
|
||||
def resolve_request_delay() -> float:
|
||||
"""Read REQUEST_DELAY from the environment, defaulting to DEFAULT_REQUEST_DELAY."""
|
||||
raw = os.getenv("REQUEST_DELAY", "").strip()
|
||||
if not raw:
|
||||
return DEFAULT_REQUEST_DELAY
|
||||
try:
|
||||
value = float(raw)
|
||||
except ValueError:
|
||||
logger.warning(
|
||||
"REQUEST_DELAY=%r is not a number — using default %.1fs.",
|
||||
raw,
|
||||
DEFAULT_REQUEST_DELAY,
|
||||
)
|
||||
return DEFAULT_REQUEST_DELAY
|
||||
if value < 0:
|
||||
logger.warning(
|
||||
"REQUEST_DELAY=%r is negative — using default %.1fs.",
|
||||
raw,
|
||||
DEFAULT_REQUEST_DELAY,
|
||||
)
|
||||
return DEFAULT_REQUEST_DELAY
|
||||
return value
|
||||
|
||||
# Realistic Chrome User-Agent
|
||||
USER_AGENT = (
|
||||
"Mozilla/5.0 (X11; Linux x86_64) "
|
||||
@@ -75,6 +133,9 @@ class BaseProvider(ABC):
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
}
|
||||
)
|
||||
# Polite pacing between consecutive requests (REQUEST_DELAY env).
|
||||
self._request_delay = resolve_request_delay()
|
||||
self._last_request_at: float | None = None
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Abstract interface — subclasses must implement these
|
||||
@@ -89,13 +150,36 @@ class BaseProvider(ABC):
|
||||
"""Return the full conversation detail for a single ID."""
|
||||
|
||||
@abstractmethod
|
||||
def normalize_conversation(self, raw: dict) -> dict:
|
||||
"""Transform provider-specific schema to the common normalized schema."""
|
||||
def normalize_conversation(self, raw: dict, loss_report: LossReport | None = None) -> dict:
|
||||
"""Transform provider-specific schema to the common normalized schema.
|
||||
|
||||
``loss_report`` accumulates counts of dropped/unhandled content so the
|
||||
export loop can surface a single summary at the end. When None, providers
|
||||
construct a throwaway local report (so calling normalize_conversation in
|
||||
isolation, e.g. from tests, doesn't crash).
|
||||
"""
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Concrete helpers
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _pace(self) -> None:
|
||||
"""Sleep so consecutive requests are at least REQUEST_DELAY apart (±25% jitter).
|
||||
|
||||
getattr fallback: tests construct providers via ``__new__``, skipping
|
||||
``__init__`` — those run unpaced.
|
||||
"""
|
||||
delay = getattr(self, "_request_delay", 0)
|
||||
if delay <= 0:
|
||||
return
|
||||
target = delay * random.uniform(0.75, 1.25)
|
||||
last = getattr(self, "_last_request_at", None)
|
||||
if last is not None:
|
||||
wait = target - (time.monotonic() - last)
|
||||
if wait > 0:
|
||||
time.sleep(wait)
|
||||
self._last_request_at = time.monotonic()
|
||||
|
||||
def fetch_all_conversations(self, since: datetime | None = None) -> list[dict]:
|
||||
"""Fetch every conversation, handling pagination automatically.
|
||||
|
||||
@@ -201,6 +285,7 @@ class BaseProvider(ABC):
|
||||
|
||||
while attempt <= MAX_RETRIES:
|
||||
attempt += 1
|
||||
self._pace()
|
||||
start = time.monotonic()
|
||||
|
||||
try:
|
||||
|
||||
+695
-85
@@ -19,13 +19,40 @@ Response: {"items": [...], "cursor": "<opaque_base64_or_null>"}
|
||||
Pagination ends when cursor is null or an empty string.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from curl_cffi import requests as curl_requests
|
||||
|
||||
from src.providers.base import BaseProvider, ProviderError, REQUEST_TIMEOUT
|
||||
from src.blocks import (
|
||||
COLLAPSED_KIND_HIDDEN_CONTEXT,
|
||||
COLLAPSED_KIND_TOOL_DUMP,
|
||||
UNKNOWN_REASON_EXTRACTION_FAILED,
|
||||
UNKNOWN_REASON_UNKNOWN_FIELD_IN_KNOWN_TYPE,
|
||||
UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
make_code_block,
|
||||
make_collapsed_block,
|
||||
make_file_placeholder,
|
||||
make_hidden_context_marker,
|
||||
make_image_placeholder,
|
||||
make_text_block,
|
||||
make_thinking_block,
|
||||
make_tool_result_block,
|
||||
make_unknown_block,
|
||||
)
|
||||
from src.loss_report import LossReport
|
||||
from src.providers.base import (
|
||||
BaseProvider,
|
||||
HIDDEN_CONTENT_FULL,
|
||||
HIDDEN_CONTENT_OMIT,
|
||||
HIDDEN_CONTENT_PLACEHOLDER,
|
||||
ProviderError,
|
||||
REQUEST_TIMEOUT,
|
||||
VALID_HIDDEN_CONTENT_POLICIES,
|
||||
resolve_hidden_content_policy,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -36,6 +63,34 @@ AUTH_SESSION_URL = "https://chatgpt.com/api/auth/session"
|
||||
# Run: python -c "from curl_cffi.requests import BrowserType; print(list(BrowserType))"
|
||||
IMPERSONATE = "chrome120"
|
||||
|
||||
def parse_asset_file_id(ref: str) -> str | None:
|
||||
"""Extract the file ID from a ChatGPT asset pointer.
|
||||
|
||||
Observed forms (2026-06-12):
|
||||
sediment://file_00000000245c71fda50034d7f1647791 → file_…
|
||||
sediment://8456107fc383a53#file_…979c…#p_6.png (generated) → file_…
|
||||
file-service://file-AbCdEf → file-AbCdEf
|
||||
"""
|
||||
if not isinstance(ref, str) or not ref:
|
||||
return None
|
||||
if ref.startswith("file-service://"):
|
||||
return ref.removeprefix("file-service://") or None
|
||||
if ref.startswith("sediment://"):
|
||||
body = ref.removeprefix("sediment://")
|
||||
for part in body.split("#"):
|
||||
if part.startswith("file_") or part.startswith("file-"):
|
||||
return part
|
||||
return body or None
|
||||
return None
|
||||
|
||||
|
||||
# Tool-role authors whose messages are retrieval dumps — the full text of
|
||||
# attached/project files re-injected by ChatGPT on every tool run. Measured
|
||||
# 2026-06-12 across this user's three largest conversations: 86% of all
|
||||
# content bytes, not flagged is_visually_hidden_from_conversation.
|
||||
# myfiles_browser is the legacy name for the same retrieval tool.
|
||||
_COLLAPSE_TOOL_AUTHORS = {"file_search", "myfiles_browser"}
|
||||
|
||||
|
||||
class ChatGPTProvider(BaseProvider):
|
||||
"""Provider for ChatGPT conversations via the internal web API.
|
||||
@@ -58,6 +113,7 @@ class ChatGPTProvider(BaseProvider):
|
||||
session_token: str | None = None,
|
||||
session_token_1: str | None = None,
|
||||
project_ids: list[str] | None = None,
|
||||
hidden_content: str | None = None,
|
||||
) -> None:
|
||||
# Pass a curl_cffi session to the base class instead of a requests.Session.
|
||||
# curl_cffi.requests.Session is API-compatible with requests.Session.
|
||||
@@ -98,6 +154,14 @@ class ChatGPTProvider(BaseProvider):
|
||||
# Cache of project_id → display name (avoids re-fetching gizmo details)
|
||||
self._project_name_cache: dict[str, str] = {}
|
||||
|
||||
# Policy for messages invisible in the web UI (retrieval dumps,
|
||||
# hidden-flagged context): full | placeholder | omit.
|
||||
self._hidden_content = (
|
||||
hidden_content
|
||||
if hidden_content in VALID_HIDDEN_CONTENT_POLICIES
|
||||
else resolve_hidden_content_policy()
|
||||
)
|
||||
|
||||
# ChatGPT now splits large session cookies into .0 / .1 chunks.
|
||||
# Always send both named chunks; the server reassembles them.
|
||||
self._session.cookies.set(
|
||||
@@ -547,11 +611,64 @@ class ChatGPTProvider(BaseProvider):
|
||||
)
|
||||
return data
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Asset downloads
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def download_asset(self, ref: str) -> tuple[bytes, str | None, str | None]:
|
||||
"""Download a sediment:// / file-service:// asset.
|
||||
|
||||
Two hops (verified live 2026-06-12): the files endpoint returns a
|
||||
signed ``download_url``; fetching that yields the bytes. Expired
|
||||
assets (e.g. old generated images) 404 on the first hop.
|
||||
|
||||
Returns:
|
||||
(content_bytes, mime_type_or_None, file_name_or_None)
|
||||
|
||||
Raises:
|
||||
ProviderError: unparseable ref, expired/missing asset, or
|
||||
download failure.
|
||||
"""
|
||||
file_id = parse_asset_file_id(ref)
|
||||
if not file_id:
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"download_asset({ref[:40]})",
|
||||
ValueError(f"Unrecognised asset reference: {ref[:80]}"),
|
||||
)
|
||||
|
||||
meta = self._make_request("GET", f"{BASE_URL}/files/{file_id}/download")
|
||||
download_url = meta.get("download_url")
|
||||
if not download_url:
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"download_asset({file_id})",
|
||||
RuntimeError(f"No download_url in response: {meta.get('detail') or meta}"),
|
||||
)
|
||||
|
||||
self._pace()
|
||||
resp = self._session.request("GET", download_url, timeout=REQUEST_TIMEOUT)
|
||||
if resp.status_code != 200:
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"download_asset({file_id})",
|
||||
RuntimeError(f"Signed URL returned HTTP {resp.status_code}"),
|
||||
)
|
||||
mime = resp.headers.get("content-type") or None
|
||||
file_name = meta.get("file_name") or None
|
||||
logger.debug(
|
||||
"[chatgpt] Downloaded asset %s: %d bytes (%s)",
|
||||
file_id,
|
||||
len(resp.content),
|
||||
mime,
|
||||
)
|
||||
return resp.content, mime, file_name
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Normalization
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def normalize_conversation(self, raw: dict) -> dict:
|
||||
def normalize_conversation(self, raw: dict, loss_report: LossReport | None = None) -> dict:
|
||||
"""Transform ChatGPT raw schema to the common normalized schema.
|
||||
|
||||
ChatGPT stores messages in a nested ``mapping`` dict where each node
|
||||
@@ -562,7 +679,11 @@ class ChatGPTProvider(BaseProvider):
|
||||
fetch_all_conversations). The conversation detail endpoint does not
|
||||
include project information.
|
||||
"""
|
||||
conv_id = raw.get("id", "")
|
||||
report = loss_report if loss_report is not None else LossReport()
|
||||
# ChatGPT's /backend-api/conversation/<id> response uses ``conversation_id``
|
||||
# at the top level (not ``id``); fixtures and listing summaries use ``id``.
|
||||
# Read both so both code paths populate the normalized ``id`` correctly.
|
||||
conv_id = raw.get("id") or raw.get("conversation_id") or ""
|
||||
title = raw.get("title") or "Untitled"
|
||||
created_at = _ts_to_iso(raw.get("create_time"))
|
||||
updated_at = _ts_to_iso(raw.get("update_time"))
|
||||
@@ -580,7 +701,12 @@ class ChatGPTProvider(BaseProvider):
|
||||
)
|
||||
|
||||
mapping: dict = raw.get("mapping", {})
|
||||
messages = _extract_messages(mapping, raw, conv_id)
|
||||
# getattr fallback: tests construct providers via __new__, skipping __init__.
|
||||
policy = getattr(self, "_hidden_content", None) or resolve_hidden_content_policy()
|
||||
messages = _extract_messages(mapping, raw, conv_id, report, policy)
|
||||
for _ in messages:
|
||||
report.record_message()
|
||||
report.record_conversation()
|
||||
|
||||
return {
|
||||
"id": conv_id,
|
||||
@@ -610,14 +736,22 @@ def _ts_to_iso(ts: float | int | str | None) -> str:
|
||||
|
||||
|
||||
def _extract_messages(
|
||||
mapping: dict[str, Any], raw: dict, conv_id: str
|
||||
mapping: dict[str, Any],
|
||||
raw: dict,
|
||||
conv_id: str,
|
||||
report: LossReport,
|
||||
policy: str = HIDDEN_CONTENT_FULL,
|
||||
) -> list[dict]:
|
||||
"""Walk the ChatGPT conversation mapping tree to produce an ordered message list."""
|
||||
"""Walk the ChatGPT conversation mapping tree to produce an ordered message list.
|
||||
|
||||
All roles (user/assistant/system/tool) are processed; the prior filter that
|
||||
dropped non-user/assistant messages is lifted in v0.4.0 — truly empty
|
||||
messages skip via the empty-content guard, anything with content renders.
|
||||
"""
|
||||
if not mapping:
|
||||
logger.warning("[chatgpt] Conversation %s has empty mapping", conv_id[:8])
|
||||
return []
|
||||
|
||||
# Find the root node (the one that has no parent, or whose parent is None/not in mapping)
|
||||
root_id = _find_root(mapping)
|
||||
if root_id is None:
|
||||
logger.warning(
|
||||
@@ -635,68 +769,12 @@ def _extract_messages(
|
||||
|
||||
node = mapping.get(node_id, {})
|
||||
msg_data = node.get("message")
|
||||
|
||||
if msg_data:
|
||||
role = msg_data.get("author", {}).get("role", "")
|
||||
# Skip system/tool messages silently unless they have visible content
|
||||
if role in ("user", "assistant"):
|
||||
content_obj = msg_data.get("content", {})
|
||||
content_type = content_obj.get("content_type", "text")
|
||||
ts = msg_data.get("create_time")
|
||||
built = _build_message(msg_data, conv_id, node_id, report, policy)
|
||||
if built is not None:
|
||||
messages.append(built)
|
||||
|
||||
# Content types whose parts[] contain plain text strings.
|
||||
# model_editable_context / user_editable_context = project instructions
|
||||
# thoughts / reasoning_recap = o1/o3 reasoning traces
|
||||
_TEXT_PARTS_TYPES = {
|
||||
"text",
|
||||
"model_editable_context",
|
||||
"user_editable_context",
|
||||
"thoughts",
|
||||
"reasoning_recap",
|
||||
}
|
||||
|
||||
if content_type in _TEXT_PARTS_TYPES:
|
||||
text = _extract_text(content_obj, conv_id, node_id)
|
||||
if text:
|
||||
messages.append(
|
||||
{
|
||||
"role": role,
|
||||
"content": text,
|
||||
"content_type": "text",
|
||||
"timestamp": _ts_to_iso(ts) if ts else None,
|
||||
}
|
||||
)
|
||||
else:
|
||||
logger.debug(
|
||||
"[chatgpt] Skipping empty %s message in conversation %s",
|
||||
content_type,
|
||||
conv_id[:8],
|
||||
)
|
||||
elif content_type == "code":
|
||||
# Inline code response — extract and wrap in a fenced code block
|
||||
code_text = content_obj.get("text") or "\n".join(
|
||||
p for p in content_obj.get("parts", []) if isinstance(p, str)
|
||||
)
|
||||
language = content_obj.get("language", "")
|
||||
if code_text:
|
||||
messages.append(
|
||||
{
|
||||
"role": role,
|
||||
"content": f"```{language}\n{code_text}\n```",
|
||||
"content_type": "code",
|
||||
"timestamp": _ts_to_iso(ts) if ts else None,
|
||||
}
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"[chatgpt] Skipping %s content in conversation %s message %s "
|
||||
"— rich content not yet supported (see FUTURE.md)",
|
||||
content_type,
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
|
||||
# Walk children in order (ChatGPT typically has one child per node in a linear chat)
|
||||
# Walk children in order (linear in typical conversations)
|
||||
for child_id in node.get("children", []):
|
||||
walk(child_id)
|
||||
|
||||
@@ -718,36 +796,568 @@ def _find_root(mapping: dict[str, Any]) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def _extract_text(content_obj: dict, conv_id: str, node_id: str) -> str:
|
||||
"""Extract plain text from a ChatGPT content object.
|
||||
def _build_message(
|
||||
msg_data: dict,
|
||||
conv_id: str,
|
||||
node_id: str,
|
||||
report: LossReport,
|
||||
policy: str = HIDDEN_CONTENT_FULL,
|
||||
) -> dict | None:
|
||||
"""Construct a normalized message dict (with ``blocks``) for one ChatGPT node.
|
||||
|
||||
Handles three part shapes:
|
||||
- str — plain text (most messages)
|
||||
- dict with content_type="text" — wrapped text part
|
||||
- dict with "content" key — o1/o3 thoughts/reasoning parts
|
||||
Returns None for messages that should be skipped (truly empty, or omitted
|
||||
by the EXPORTER_HIDDEN_CONTENT policy). Otherwise returns a dict with
|
||||
``role``, ``content_type``, ``timestamp``, ``blocks``.
|
||||
"""
|
||||
parts = content_obj.get("parts", [])
|
||||
if not parts:
|
||||
return ""
|
||||
author = msg_data.get("author") or {}
|
||||
role = author.get("role", "") or ""
|
||||
if role not in ("user", "assistant", "system", "tool"):
|
||||
# Unrecognised role — log and surface, but pass through so role metadata
|
||||
# is preserved for the reader.
|
||||
logger.debug(
|
||||
"[chatgpt] Unrecognised role %r in conversation %s message %s",
|
||||
role,
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
|
||||
content_obj = msg_data.get("content") or {}
|
||||
content_type = content_obj.get("content_type", "text")
|
||||
ts = msg_data.get("create_time")
|
||||
metadata = msg_data.get("metadata") or {}
|
||||
is_hidden = bool(metadata.get("is_visually_hidden_from_conversation"))
|
||||
author_name = author.get("name") or None
|
||||
|
||||
blocks = _extract_blocks_for_content(
|
||||
content_type, content_obj, role, conv_id, node_id, report,
|
||||
author_name=author_name, msg_metadata=metadata,
|
||||
)
|
||||
|
||||
if not blocks:
|
||||
logger.debug(
|
||||
"[chatgpt] Skipping empty %s message in conversation %s",
|
||||
content_type,
|
||||
conv_id[:8],
|
||||
)
|
||||
return None
|
||||
|
||||
# Messages invisible in the web UI: tool retrieval dumps (identified by
|
||||
# author.name — they are NOT hidden-flagged) and hidden-flagged context.
|
||||
collapse_kind: str | None = None
|
||||
if role == "tool" and author_name in _COLLAPSE_TOOL_AUTHORS:
|
||||
collapse_kind = COLLAPSED_KIND_TOOL_DUMP
|
||||
elif is_hidden:
|
||||
collapse_kind = COLLAPSED_KIND_HIDDEN_CONTEXT
|
||||
|
||||
if collapse_kind is not None and policy != HIDDEN_CONTENT_FULL:
|
||||
origin = (
|
||||
author_name if collapse_kind == COLLAPSED_KIND_TOOL_DUMP else content_type
|
||||
) or "?"
|
||||
size_bytes = len(json.dumps(content_obj, ensure_ascii=False, default=str))
|
||||
report.record_collapsed(origin, size_bytes)
|
||||
logger.debug(
|
||||
"[chatgpt] %s %s message (%s, %dB) in conversation %s "
|
||||
"per EXPORTER_HIDDEN_CONTENT=%s",
|
||||
"Omitted" if policy == HIDDEN_CONTENT_OMIT else "Collapsed",
|
||||
collapse_kind,
|
||||
origin,
|
||||
size_bytes,
|
||||
conv_id[:8],
|
||||
policy,
|
||||
)
|
||||
if policy == HIDDEN_CONTENT_OMIT:
|
||||
return None
|
||||
blocks = [
|
||||
make_collapsed_block(
|
||||
origin=origin,
|
||||
content_type=content_type,
|
||||
size_bytes=size_bytes,
|
||||
kind=collapse_kind,
|
||||
)
|
||||
]
|
||||
elif is_hidden:
|
||||
# Policy 'full': prepend a marker so the reader knows this message is
|
||||
# hidden in the source UI. The marker is content-type-agnostic.
|
||||
blocks = [make_hidden_context_marker(content_type)] + blocks
|
||||
|
||||
# Vestigial content_type: "code" for code-only messages, otherwise "text"
|
||||
msg_content_type = "code" if (
|
||||
len(blocks) == 1 and blocks[0].get("type") == "code"
|
||||
) else "text"
|
||||
|
||||
return {
|
||||
"role": role or "user",
|
||||
"content_type": msg_content_type,
|
||||
"timestamp": _ts_to_iso(ts) if ts else None,
|
||||
"blocks": blocks,
|
||||
}
|
||||
|
||||
|
||||
# Content types whose ``parts`` are plain text strings.
|
||||
_PLAIN_TEXT_PARTS_TYPES = {"text"}
|
||||
# Content types that carry inline reasoning/thoughts.
|
||||
_THINKING_TYPES = {"thoughts", "reasoning_recap"}
|
||||
# Custom-Instructions / model-context types — direct fields, NOT parts.
|
||||
_DIRECT_FIELD_CONTEXT_TYPES = {
|
||||
"user_editable_context",
|
||||
"model_editable_context",
|
||||
}
|
||||
# Known direct fields per context type. Anything not listed but non-null
|
||||
# becomes an `unknown` block per the no-silent-drop-of-non-null-fields rule.
|
||||
_USER_EDITABLE_CONTEXT_KNOWN_FIELDS = ("user_profile", "user_instructions")
|
||||
_MODEL_EDITABLE_CONTEXT_KNOWN_FIELDS = (
|
||||
"model_set_context",
|
||||
"repository",
|
||||
"repo_summary",
|
||||
"structured_context",
|
||||
)
|
||||
|
||||
|
||||
def _extract_blocks_for_content(
|
||||
content_type: str,
|
||||
content_obj: dict,
|
||||
role: str,
|
||||
conv_id: str,
|
||||
node_id: str,
|
||||
report: LossReport,
|
||||
author_name: str | None = None,
|
||||
msg_metadata: dict | None = None,
|
||||
) -> list[dict]:
|
||||
"""Dispatch on content_type and return a list of blocks for one message."""
|
||||
|
||||
if content_type in _PLAIN_TEXT_PARTS_TYPES:
|
||||
return _extract_text_content_type_blocks(content_obj, conv_id, node_id, report)
|
||||
|
||||
if content_type == "multimodal_text":
|
||||
return _extract_multimodal_blocks(content_obj, role, conv_id, node_id, report)
|
||||
|
||||
if content_type == "execution_output":
|
||||
return _extract_execution_output_blocks(
|
||||
content_obj, author_name, msg_metadata or {}, conv_id, node_id
|
||||
)
|
||||
|
||||
if content_type == "system_error":
|
||||
return _extract_system_error_blocks(content_obj, author_name)
|
||||
|
||||
if content_type == "tether_browsing_display":
|
||||
return _extract_tether_browsing_display_blocks(
|
||||
content_obj, author_name, conv_id, node_id
|
||||
)
|
||||
|
||||
if content_type == "code":
|
||||
code_text = content_obj.get("text") or "\n".join(
|
||||
p for p in content_obj.get("parts", []) if isinstance(p, str)
|
||||
)
|
||||
language = content_obj.get("language", "") or ""
|
||||
block = make_code_block(code_text, language)
|
||||
return [block] if block else []
|
||||
|
||||
if content_type in _THINKING_TYPES:
|
||||
text = _join_string_parts(content_obj)
|
||||
block = make_thinking_block(text)
|
||||
return [block] if block else []
|
||||
|
||||
if content_type in _DIRECT_FIELD_CONTEXT_TYPES:
|
||||
return _extract_editable_context_blocks(
|
||||
content_type, content_obj, conv_id, node_id, report
|
||||
)
|
||||
|
||||
if content_type == "image_asset_pointer":
|
||||
# Top-level image (rare — usually nested inside multimodal_text).
|
||||
ref = content_obj.get("asset_pointer", "")
|
||||
source = "user_upload" if role == "user" else "model_generated"
|
||||
return [make_image_placeholder(ref=ref, source=source)]
|
||||
|
||||
# Unknown content_type → visible unknown block + WARNING + tally
|
||||
keys = list(content_obj.keys())
|
||||
logger.warning(
|
||||
"[chatgpt] Unknown content_type %r in conversation %s message %s "
|
||||
"— see plan §Data-loss visibility (rendering as unknown block)",
|
||||
content_type,
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
report.record_unknown(content_type or "?")
|
||||
return [
|
||||
make_unknown_block(
|
||||
raw_type=content_type or "?",
|
||||
observed_keys=keys,
|
||||
reason=UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _extract_text_content_type_blocks(
|
||||
content_obj: dict, conv_id: str, node_id: str, report: LossReport
|
||||
) -> list[dict]:
|
||||
"""Extract blocks for ``content_type == "text"``.
|
||||
|
||||
Plural-parts rule: emit ONE text block per message with all string parts
|
||||
joined by ``\\n``. Don't emit one block per part.
|
||||
|
||||
Dict parts inside a text content_type message (the suspected o1/o3 reasoning
|
||||
subpart shape ``{"summary": ..., "content": ...}``) are preserved as text
|
||||
today — defensive behavior pending real-data capture in v0.4.1.
|
||||
"""
|
||||
parts = content_obj.get("parts", []) or []
|
||||
string_chunks: list[str] = []
|
||||
|
||||
text_parts = []
|
||||
for part in parts:
|
||||
if isinstance(part, str):
|
||||
text_parts.append(part)
|
||||
string_chunks.append(part)
|
||||
elif isinstance(part, dict):
|
||||
part_type = part.get("content_type", "")
|
||||
if part_type == "text":
|
||||
text_parts.append(part.get("text", ""))
|
||||
txt = part.get("text", "") or ""
|
||||
if txt:
|
||||
string_chunks.append(txt)
|
||||
elif "content" in part:
|
||||
# o1/o3 thoughts parts: {"summary": "...", "content": "..."}
|
||||
text_parts.append(part["content"])
|
||||
# Suspected o1/o3 reasoning subpart. Defensive: preserve as text
|
||||
# block (matches current behavior). v0.4.1 reclassifies once
|
||||
# the real shape is captured live.
|
||||
content_val = part.get("content", "") or ""
|
||||
if content_val:
|
||||
string_chunks.append(content_val)
|
||||
elif part_type:
|
||||
# Image, file, or other binary attachment — skip and warn
|
||||
# Non-text dict part inside a text content_type — surface it.
|
||||
logger.warning(
|
||||
"[chatgpt] Skipping %s attachment in conversation %s "
|
||||
"— rich content not yet supported (see FUTURE.md)",
|
||||
"[chatgpt] Unexpected %s part inside text content_type "
|
||||
"in conversation %s message %s — rendering as unknown block",
|
||||
part_type,
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
report.record_unknown(part_type)
|
||||
# Inline mark in the joined text so order is preserved.
|
||||
string_chunks.append(
|
||||
f"\n[Unknown part: type={part_type}; "
|
||||
f"keys={list(part.keys())[:10]}]\n"
|
||||
)
|
||||
|
||||
return "\n".join(t for t in text_parts if t)
|
||||
joined = "\n".join(c for c in string_chunks if c)
|
||||
block = make_text_block(joined)
|
||||
return [block] if block else []
|
||||
|
||||
|
||||
def _join_string_parts(content_obj: dict) -> str:
|
||||
"""Helper: join all string parts in ``parts`` with newlines."""
|
||||
parts = content_obj.get("parts", []) or []
|
||||
return "\n".join(p for p in parts if isinstance(p, str) and p)
|
||||
|
||||
|
||||
def _extract_multimodal_blocks(
|
||||
content_obj: dict, role: str, conv_id: str, node_id: str, report: LossReport
|
||||
) -> list[dict]:
|
||||
"""Extract blocks from a ``multimodal_text`` content object.
|
||||
|
||||
Walks ``parts`` in array order — order varies between user and assistant
|
||||
turns, and the extractor preserves source ordering. Emits text +
|
||||
image_placeholder + file_placeholder blocks per part.
|
||||
"""
|
||||
parts = content_obj.get("parts", []) or []
|
||||
blocks: list[dict] = []
|
||||
|
||||
for part in parts:
|
||||
if isinstance(part, str):
|
||||
block = make_text_block(part)
|
||||
if block:
|
||||
blocks.append(block)
|
||||
continue
|
||||
|
||||
if not isinstance(part, dict):
|
||||
continue
|
||||
|
||||
part_type = part.get("content_type", "")
|
||||
|
||||
if part_type == "audio_transcription":
|
||||
txt = part.get("text", "") or ""
|
||||
block = make_text_block(txt)
|
||||
if block:
|
||||
blocks.append(block)
|
||||
elif "text" not in part:
|
||||
logger.warning(
|
||||
"[chatgpt] audio_transcription part missing 'text' key "
|
||||
"in conversation %s message %s",
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
report.record_extraction_failure("audio_transcription")
|
||||
blocks.append(
|
||||
make_unknown_block(
|
||||
raw_type="audio_transcription",
|
||||
observed_keys=list(part.keys()),
|
||||
reason=UNKNOWN_REASON_EXTRACTION_FAILED,
|
||||
summary="expected key 'text' not found",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
if part_type == "image_asset_pointer":
|
||||
ref = part.get("asset_pointer", "")
|
||||
source = "user_upload" if role == "user" else "model_generated"
|
||||
mime = None
|
||||
blocks.append(make_image_placeholder(ref=ref, source=source, mime=mime))
|
||||
continue
|
||||
|
||||
if part_type == "audio_asset_pointer":
|
||||
blocks.append(_audio_asset_placeholder(part))
|
||||
continue
|
||||
|
||||
if part_type == "real_time_user_audio_video_asset_pointer":
|
||||
# Wrapper carrying a nested audio_asset_pointer + optional video frames.
|
||||
nested_audio = part.get("audio_asset_pointer")
|
||||
if isinstance(nested_audio, dict):
|
||||
blocks.append(_audio_asset_placeholder(nested_audio))
|
||||
else:
|
||||
logger.warning(
|
||||
"[chatgpt] real_time_user_audio_video_asset_pointer missing "
|
||||
"nested audio_asset_pointer in conversation %s message %s",
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
report.record_extraction_failure(
|
||||
"real_time_user_audio_video_asset_pointer"
|
||||
)
|
||||
blocks.append(
|
||||
make_unknown_block(
|
||||
raw_type="real_time_user_audio_video_asset_pointer",
|
||||
observed_keys=list(part.keys()),
|
||||
reason=UNKNOWN_REASON_EXTRACTION_FAILED,
|
||||
summary="expected nested 'audio_asset_pointer' not found",
|
||||
)
|
||||
)
|
||||
|
||||
frames = part.get("frames_asset_pointers") or []
|
||||
if frames:
|
||||
# Defensive: empty in all observed cases, but if non-empty
|
||||
# surface as a separate file placeholder.
|
||||
video_ref = part.get("video_container_asset_pointer") or "(video frames)"
|
||||
blocks.append(
|
||||
make_file_placeholder(
|
||||
ref=str(video_ref),
|
||||
mime="video/unknown",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
# Anything else inside multimodal_text — visible unknown block
|
||||
logger.warning(
|
||||
"[chatgpt] Unknown multimodal_text part type %r in conversation %s message %s",
|
||||
part_type,
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
report.record_unknown(part_type or "?")
|
||||
blocks.append(
|
||||
make_unknown_block(
|
||||
raw_type=part_type or "?",
|
||||
observed_keys=list(part.keys()),
|
||||
reason=UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
)
|
||||
)
|
||||
|
||||
return blocks
|
||||
|
||||
|
||||
def _audio_asset_placeholder(audio_part: dict) -> dict:
|
||||
"""Build a file_placeholder for an audio_asset_pointer dict.
|
||||
|
||||
Handles missing/zero metadata defensively.
|
||||
"""
|
||||
ref = audio_part.get("asset_pointer", "") or ""
|
||||
fmt = audio_part.get("format") or "unknown"
|
||||
size_bytes = audio_part.get("size_bytes")
|
||||
if not isinstance(size_bytes, int) or size_bytes <= 0:
|
||||
size_bytes = None
|
||||
metadata = audio_part.get("metadata") or {}
|
||||
start = metadata.get("start") if isinstance(metadata, dict) else None
|
||||
end = metadata.get("end") if isinstance(metadata, dict) else None
|
||||
duration: float | None = None
|
||||
if isinstance(start, (int, float)) and isinstance(end, (int, float)):
|
||||
diff = float(end) - float(start)
|
||||
if diff > 0:
|
||||
duration = diff
|
||||
return make_file_placeholder(
|
||||
ref=ref,
|
||||
mime=f"audio/{fmt}" if fmt else "audio/unknown",
|
||||
size_bytes=size_bytes,
|
||||
duration_seconds=duration,
|
||||
)
|
||||
|
||||
|
||||
def _extract_editable_context_blocks(
|
||||
content_type: str, content_obj: dict, conv_id: str, node_id: str, report: LossReport
|
||||
) -> list[dict]:
|
||||
"""Extract blocks from user_editable_context / model_editable_context messages.
|
||||
|
||||
These have no ``parts`` field — they carry direct keys. Read all known
|
||||
fields, emit one labeled fenced block per non-null known field, and emit an
|
||||
``unknown`` block for any unrecognised non-null direct field (no-silent-drop
|
||||
rule).
|
||||
"""
|
||||
if content_type == "user_editable_context":
|
||||
known_fields: tuple[str, ...] = _USER_EDITABLE_CONTEXT_KNOWN_FIELDS
|
||||
elif content_type == "model_editable_context":
|
||||
known_fields = _MODEL_EDITABLE_CONTEXT_KNOWN_FIELDS
|
||||
else:
|
||||
known_fields = ()
|
||||
|
||||
blocks: list[dict] = []
|
||||
label_kind = "Custom Instructions" if content_type == "user_editable_context" else "Model Context"
|
||||
|
||||
for field in known_fields:
|
||||
value = content_obj.get(field)
|
||||
if value is None or (isinstance(value, str) and not value.strip()):
|
||||
continue
|
||||
if isinstance(value, (dict, list)):
|
||||
# Render as a JSON-rendered text block. _safe_fence will wrap it.
|
||||
import json as _json
|
||||
rendered = _json.dumps(value, indent=2, default=str, ensure_ascii=False)
|
||||
else:
|
||||
rendered = str(value)
|
||||
label = f"**{label_kind} — {field}:**"
|
||||
# Emit as text block; the renderer's _safe_fence wraps the raw value.
|
||||
# We use a "labeled fenced block" pattern: header line + raw content
|
||||
# joined inside one text block, where the renderer will leave it alone.
|
||||
# To get the safe-fence wrap we use a code block (which calls _safe_fence
|
||||
# internally and renders without language-hint corruption risk).
|
||||
blocks.append(make_text_block(label))
|
||||
code_block = make_code_block(rendered, language="")
|
||||
if code_block:
|
||||
blocks.append(code_block)
|
||||
|
||||
# Catch unknown non-null direct fields (no-silent-drop rule).
|
||||
structural_keys = {"content_type", "parts"}
|
||||
for key, value in content_obj.items():
|
||||
if key in structural_keys or key in known_fields:
|
||||
continue
|
||||
if value is None:
|
||||
continue
|
||||
# Reject null/empty containers.
|
||||
if isinstance(value, (str, list, dict)) and not value:
|
||||
continue
|
||||
logger.warning(
|
||||
"[chatgpt] Unknown non-null field %r in %s message %s/%s",
|
||||
key,
|
||||
content_type,
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
report.record_unknown(f"{content_type}.{key}")
|
||||
blocks.append(
|
||||
make_unknown_block(
|
||||
raw_type=f"{content_type}.{key}",
|
||||
observed_keys=list(content_obj.keys()),
|
||||
reason=UNKNOWN_REASON_UNKNOWN_FIELD_IN_KNOWN_TYPE,
|
||||
summary=f"unknown non-null field '{key}' in {content_type}",
|
||||
)
|
||||
)
|
||||
|
||||
return blocks
|
||||
|
||||
|
||||
def _extract_execution_output_blocks(
|
||||
content_obj: dict,
|
||||
author_name: str | None,
|
||||
msg_metadata: dict,
|
||||
conv_id: str,
|
||||
node_id: str,
|
||||
) -> list[dict]:
|
||||
"""Map a ChatGPT ``execution_output`` content (Code Interpreter / container.exec
|
||||
/ python tool output) onto a ``tool_result`` block.
|
||||
|
||||
Locked shape (captured live during planning v0.4.1):
|
||||
content.text → output
|
||||
author.name → tool_name
|
||||
metadata.aggregate_result.status → "error" → is_error=True
|
||||
metadata.reasoning_title → summary
|
||||
|
||||
Empty ``content.text`` → skip (DEBUG log) — a tool that emits no output is
|
||||
a transient artifact, not archival content.
|
||||
"""
|
||||
text = content_obj.get("text") or ""
|
||||
if not text.strip():
|
||||
logger.debug(
|
||||
"[chatgpt] Skipping empty execution_output in conversation %s message %s",
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
return []
|
||||
|
||||
aggregate = msg_metadata.get("aggregate_result") or {}
|
||||
status = aggregate.get("status") if isinstance(aggregate, dict) else None
|
||||
is_error = isinstance(status, str) and status.lower() == "error"
|
||||
summary = msg_metadata.get("reasoning_title") or None
|
||||
|
||||
return [
|
||||
make_tool_result_block(
|
||||
output=text,
|
||||
tool_name=author_name,
|
||||
is_error=is_error,
|
||||
summary=summary,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _extract_system_error_blocks(
|
||||
content_obj: dict,
|
||||
author_name: str | None,
|
||||
) -> list[dict]:
|
||||
"""Map a ChatGPT ``system_error`` content onto an error ``tool_result`` block.
|
||||
|
||||
Captured shape: ``{content_type, name, text}`` where ``text`` is the error
|
||||
message (e.g. ``"Error: Error from browse service: 503"``). ``author.name``
|
||||
identifies the failing tool (e.g. ``"web"``).
|
||||
"""
|
||||
text = content_obj.get("text") or ""
|
||||
if not text:
|
||||
text = "(error with no message)"
|
||||
return [
|
||||
make_tool_result_block(
|
||||
output=text,
|
||||
tool_name=author_name,
|
||||
is_error=True,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _extract_tether_browsing_display_blocks(
|
||||
content_obj: dict,
|
||||
author_name: str | None,
|
||||
conv_id: str,
|
||||
node_id: str,
|
||||
) -> list[dict]:
|
||||
"""Handle ChatGPT's ``tether_browsing_display`` content.
|
||||
|
||||
Captured live: most instances are **spinner placeholders** (transient UI
|
||||
state — empty fields, ``metadata.command == "spinner"``). The actual
|
||||
retrieval content arrives as a sibling/child ``multimodal_text`` message
|
||||
that already extracts cleanly via the existing handler.
|
||||
|
||||
Locked behavior:
|
||||
- If ``result`` AND ``summary`` are both empty → skip silently (DEBUG).
|
||||
These are spinners; the real content is elsewhere.
|
||||
- Otherwise (defensive: never observed populated in real data) → render
|
||||
as a ``tool_result`` block carrying ``result`` as output and
|
||||
``summary`` as the optional summary line.
|
||||
"""
|
||||
result = content_obj.get("result") or ""
|
||||
summary = content_obj.get("summary") or ""
|
||||
|
||||
if not result.strip() and not summary.strip():
|
||||
logger.debug(
|
||||
"[chatgpt] Skipping tether_browsing_display spinner in "
|
||||
"conversation %s message %s (empty result/summary)",
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
return []
|
||||
|
||||
return [
|
||||
make_tool_result_block(
|
||||
output=result or summary,
|
||||
tool_name=author_name,
|
||||
is_error=False,
|
||||
summary=summary if result and summary else None,
|
||||
)
|
||||
]
|
||||
|
||||
+147
-47
@@ -5,6 +5,17 @@ import os
|
||||
|
||||
from curl_cffi import requests as curl_requests
|
||||
|
||||
from src.blocks import (
|
||||
UNKNOWN_REASON_EXTRACTION_FAILED,
|
||||
UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
make_image_placeholder,
|
||||
make_text_block,
|
||||
make_thinking_block,
|
||||
make_tool_result_block,
|
||||
make_tool_use_block,
|
||||
make_unknown_block,
|
||||
)
|
||||
from src.loss_report import LossReport
|
||||
from src.providers.base import BaseProvider, ProviderError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -161,8 +172,9 @@ class ClaudeProvider(BaseProvider):
|
||||
|
||||
return data
|
||||
|
||||
def normalize_conversation(self, raw: dict) -> dict:
|
||||
def normalize_conversation(self, raw: dict, loss_report: LossReport | None = None) -> dict:
|
||||
"""Transform Claude raw schema to the common normalized schema."""
|
||||
report = loss_report if loss_report is not None else LossReport()
|
||||
conv_id = raw.get("uuid") or raw.get("id", "")
|
||||
title = raw.get("name") or raw.get("title") or "Untitled"
|
||||
created_at = raw.get("created_at") or raw.get("create_time") or ""
|
||||
@@ -178,40 +190,37 @@ class ClaudeProvider(BaseProvider):
|
||||
|
||||
# Messages
|
||||
raw_messages = raw.get("chat_messages") or raw.get("messages") or []
|
||||
messages = []
|
||||
messages: list[dict] = []
|
||||
|
||||
for msg in raw_messages:
|
||||
role = _map_role(msg.get("sender") or msg.get("role", ""))
|
||||
if not role:
|
||||
continue
|
||||
|
||||
# Content can be a string or a list of content blocks
|
||||
content_raw = msg.get("content") or msg.get("text") or ""
|
||||
content, skipped_types = _extract_claude_text(content_raw, conv_id)
|
||||
|
||||
for ctype in skipped_types:
|
||||
logger.warning(
|
||||
"[claude] Skipping %s content in conversation %s "
|
||||
"— rich content not yet supported (see FUTURE.md)",
|
||||
ctype,
|
||||
conv_id[:8],
|
||||
)
|
||||
content_raw = msg.get("content") if "content" in msg else msg.get("text", "")
|
||||
blocks = _extract_claude_blocks(content_raw, conv_id, report)
|
||||
|
||||
timestamp = msg.get("created_at") or msg.get("timestamp") or None
|
||||
|
||||
if content is None:
|
||||
if not blocks:
|
||||
logger.debug("[claude] Skipping empty message in conversation %s", conv_id[:8])
|
||||
continue
|
||||
|
||||
content_type = "text"
|
||||
|
||||
messages.append(
|
||||
{
|
||||
"role": role,
|
||||
"content": content,
|
||||
"content_type": "text",
|
||||
"content_type": content_type,
|
||||
"timestamp": timestamp,
|
||||
"blocks": blocks,
|
||||
}
|
||||
)
|
||||
|
||||
for _ in messages:
|
||||
report.record_message()
|
||||
report.record_conversation()
|
||||
|
||||
return {
|
||||
"id": conv_id,
|
||||
"title": title,
|
||||
@@ -242,43 +251,134 @@ def _map_role(sender: str) -> str | None:
|
||||
return mapping.get(sender.lower()) if sender else None
|
||||
|
||||
|
||||
def _extract_claude_text(
|
||||
content: str | list | dict, conv_id: str
|
||||
) -> tuple[str | None, list[str]]:
|
||||
"""Extract plain text from a Claude content field.
|
||||
def _extract_claude_blocks(
|
||||
content: str | list | dict | None, conv_id: str, report: LossReport
|
||||
) -> list[dict]:
|
||||
"""Extract typed blocks from a Claude content field.
|
||||
|
||||
Returns:
|
||||
(text_or_None, list_of_skipped_content_types)
|
||||
Defensive dispatch — zero observed cases of rich Claude content in the
|
||||
user's archive at planning time, so this is theory-only. Real shapes
|
||||
will be locked in v0.4.1 once captured. Any unrecognised block type
|
||||
surfaces as an `unknown` block + WARNING + tally.
|
||||
"""
|
||||
skipped: list[str] = []
|
||||
if content is None:
|
||||
return []
|
||||
|
||||
if isinstance(content, str):
|
||||
text = content.strip()
|
||||
return (text if text else None), skipped
|
||||
block = make_text_block(content)
|
||||
return [block] if block else []
|
||||
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, str):
|
||||
parts.append(block)
|
||||
elif isinstance(block, dict):
|
||||
btype = block.get("type", "text")
|
||||
if btype == "text":
|
||||
t = block.get("text", "").strip()
|
||||
if t:
|
||||
parts.append(t)
|
||||
else:
|
||||
skipped.append(btype)
|
||||
text = "\n".join(parts).strip()
|
||||
return (text if text else None), skipped
|
||||
blocks: list[dict] = []
|
||||
for item in content:
|
||||
if isinstance(item, str):
|
||||
block = make_text_block(item)
|
||||
if block:
|
||||
blocks.append(block)
|
||||
elif isinstance(item, dict):
|
||||
blocks.extend(_dispatch_claude_block(item, conv_id, report))
|
||||
return blocks
|
||||
|
||||
if isinstance(content, dict):
|
||||
btype = content.get("type", "text")
|
||||
if btype == "text":
|
||||
text = content.get("text", "").strip()
|
||||
return (text if text else None), skipped
|
||||
else:
|
||||
skipped.append(btype)
|
||||
return None, skipped
|
||||
return _dispatch_claude_block(content, conv_id, report)
|
||||
|
||||
return None, skipped
|
||||
return []
|
||||
|
||||
|
||||
def _dispatch_claude_block(block: dict, conv_id: str, report: LossReport) -> list[dict]:
|
||||
"""Translate one raw Claude content block into normalized blocks."""
|
||||
btype = block.get("type", "text")
|
||||
|
||||
if btype == "text":
|
||||
block_obj = make_text_block(block.get("text", "") or "")
|
||||
return [block_obj] if block_obj else []
|
||||
|
||||
if btype == "thinking":
|
||||
# Claude extended-thinking blocks may use 'thinking' or 'text' field.
|
||||
text = block.get("thinking") or block.get("text") or ""
|
||||
block_obj = make_thinking_block(text)
|
||||
return [block_obj] if block_obj else []
|
||||
|
||||
if btype == "tool_use":
|
||||
return [
|
||||
make_tool_use_block(
|
||||
name=block.get("name", "") or "",
|
||||
input_data=block.get("input"),
|
||||
tool_id=block.get("id"),
|
||||
)
|
||||
]
|
||||
|
||||
if btype == "tool_result":
|
||||
# ``content`` may be a string or a list of nested blocks (recursive).
|
||||
nested = block.get("content")
|
||||
output = _flatten_tool_result_content(nested, conv_id, report)
|
||||
return [
|
||||
make_tool_result_block(
|
||||
output=output,
|
||||
tool_name=None,
|
||||
is_error=bool(block.get("is_error")),
|
||||
)
|
||||
]
|
||||
|
||||
if btype == "image":
|
||||
# Source shape is unverified; try the most likely fields.
|
||||
source = block.get("source") or {}
|
||||
ref = ""
|
||||
if isinstance(source, dict):
|
||||
ref = (
|
||||
source.get("file_uuid")
|
||||
or source.get("media_type")
|
||||
or source.get("url")
|
||||
or ""
|
||||
)
|
||||
return [make_image_placeholder(ref=ref or "(unknown)", source="user_upload")]
|
||||
|
||||
# Unknown block type
|
||||
keys = list(block.keys())
|
||||
logger.warning(
|
||||
"[claude] Unknown block type %r in conversation %s "
|
||||
"— see plan §Data-loss visibility (rendering as unknown block)",
|
||||
btype,
|
||||
conv_id[:8],
|
||||
)
|
||||
report.record_unknown(btype or "?")
|
||||
return [
|
||||
make_unknown_block(
|
||||
raw_type=btype or "?",
|
||||
observed_keys=keys,
|
||||
reason=UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def _flatten_tool_result_content(
|
||||
nested: object, conv_id: str, report: LossReport
|
||||
) -> str:
|
||||
"""Flatten Claude tool_result content (string OR list of nested blocks) to text.
|
||||
|
||||
Recurses into nested text blocks; any non-text nested block becomes a
|
||||
visible inline marker so non-text content isn't silently dropped.
|
||||
"""
|
||||
if nested is None:
|
||||
return ""
|
||||
if isinstance(nested, str):
|
||||
return nested
|
||||
if isinstance(nested, list):
|
||||
chunks: list[str] = []
|
||||
for item in nested:
|
||||
if isinstance(item, str):
|
||||
chunks.append(item)
|
||||
elif isinstance(item, dict):
|
||||
btype = item.get("type", "text")
|
||||
if btype == "text":
|
||||
chunks.append(item.get("text", "") or "")
|
||||
else:
|
||||
keys = list(item.keys())[:10]
|
||||
report.record_extraction_failure(f"tool_result.{btype}")
|
||||
chunks.append(
|
||||
f"[Unsupported nested {btype} block; keys={keys}]"
|
||||
)
|
||||
return "\n".join(c for c in chunks if c)
|
||||
if isinstance(nested, dict):
|
||||
return _flatten_tool_result_content([nested], conv_id, report)
|
||||
return str(nested)
|
||||
|
||||
@@ -0,0 +1,476 @@
|
||||
"""Claude Code session provider — archives local agent transcripts.
|
||||
|
||||
Reads JSONL session files from ``~/.claude/projects/<munged-cwd>/<uuid>.jsonl``
|
||||
(override with ``CLAUDE_CODE_DIR``). No tokens, no rate limits, no ToS risk —
|
||||
but the data lives in single files Claude Code may clean up, and it contains
|
||||
deliverables (reviews, plans, analyses) that exist nowhere else.
|
||||
|
||||
Rendering follows the EXPORTER_HIDDEN_CONTENT policy (decided 2026-06-12):
|
||||
prose-only by default. User prompts and assistant text are kept; tool_use /
|
||||
tool_result traffic is collapsed to one grouped placeholder per activity run
|
||||
(measured: dialogue prose is ~4% of session bytes); thinking blocks are
|
||||
dropped (counted in the run summary, no placeholder). ``full`` keeps
|
||||
everything.
|
||||
|
||||
Record types in a session file: ``user`` / ``assistant`` carry the dialogue
|
||||
(Anthropic-style ``message.content`` block arrays); ``ai-title`` carries the
|
||||
evolving session title (last one wins); ``last-prompt``,
|
||||
``file-history-snapshot``, ``attachment``, ``permission-mode``, ``system``
|
||||
are harness records and are skipped. Records flagged ``isSidechain`` are
|
||||
subagent transcripts; ``isMeta`` are harness-generated user records — both
|
||||
skipped.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from src.blocks import (
|
||||
COLLAPSED_KIND_TOOL_DUMP,
|
||||
UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
make_collapsed_block,
|
||||
make_image_placeholder,
|
||||
make_text_block,
|
||||
make_thinking_block,
|
||||
make_tool_result_block,
|
||||
make_tool_use_block,
|
||||
make_unknown_block,
|
||||
)
|
||||
from src.loss_report import LossReport
|
||||
from src.providers.base import (
|
||||
BaseProvider,
|
||||
HIDDEN_CONTENT_FULL,
|
||||
ProviderError,
|
||||
VALID_HIDDEN_CONTENT_POLICIES,
|
||||
resolve_hidden_content_policy,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_PROJECTS_DIR = "~/.claude/projects"
|
||||
|
||||
# Harness-injected tags inside user message text. Stripped so exports contain
|
||||
# the dialogue, not the CLI plumbing. A record that is nothing but tags
|
||||
# (e.g. a /model invocation) ends up empty and is skipped.
|
||||
_HARNESS_TAG_RE = re.compile(
|
||||
r"<local-command-caveat>.*?</local-command-caveat>"
|
||||
r"|<command-name>.*?</command-name>"
|
||||
r"|<command-message>.*?</command-message>"
|
||||
r"|<command-args>.*?</command-args>"
|
||||
r"|<command-contents>.*?</command-contents>"
|
||||
r"|<local-command-stdout>.*?</local-command-stdout>"
|
||||
r"|<system-reminder>.*?</system-reminder>",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
# How many distinct tool names to list in a collapsed-activity placeholder.
|
||||
_TOOL_NAMES_SHOWN = 4
|
||||
|
||||
|
||||
def _strip_harness_noise(text: str) -> str:
|
||||
if not isinstance(text, str):
|
||||
return ""
|
||||
return _HARNESS_TAG_RE.sub("", text).strip()
|
||||
|
||||
|
||||
class ClaudeCodeProvider(BaseProvider):
|
||||
"""Local-file provider over Claude Code session transcripts."""
|
||||
|
||||
provider_name = "claude-code"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
projects_dir: str | Path | None = None,
|
||||
hidden_content: str | None = None,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self._projects_dir = Path(
|
||||
projects_dir or os.getenv("CLAUDE_CODE_DIR", DEFAULT_PROJECTS_DIR)
|
||||
).expanduser()
|
||||
self._hidden_content = (
|
||||
hidden_content
|
||||
if hidden_content in VALID_HIDDEN_CONTENT_POLICIES
|
||||
else resolve_hidden_content_policy()
|
||||
)
|
||||
# conv_id → session file path, populated by _scan()
|
||||
self._path_map: dict[str, Path] = {}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# BaseProvider interface
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def list_conversations(self, offset: int = 0, limit: int = 100) -> list[dict]:
|
||||
full = self._scan()
|
||||
return full[offset : offset + limit]
|
||||
|
||||
def fetch_all_conversations(self, since: datetime | None = None) -> list[dict]:
|
||||
convs = self._scan()
|
||||
if since is not None:
|
||||
since_aware = since if since.tzinfo else since.replace(tzinfo=timezone.utc)
|
||||
convs = [
|
||||
c for c in convs
|
||||
if datetime.fromisoformat(c["updated_at"]) >= since_aware
|
||||
]
|
||||
logger.info(
|
||||
"[claude-code] Found %d session(s) under %s", len(convs), self._projects_dir
|
||||
)
|
||||
return convs
|
||||
|
||||
def get_conversation(self, conv_id: str) -> dict:
|
||||
path = self._path_map.get(conv_id)
|
||||
if path is None:
|
||||
# Direct call without a prior listing (e.g. tests) — scan first.
|
||||
self._scan()
|
||||
path = self._path_map.get(conv_id)
|
||||
if path is None or not path.exists():
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"get_conversation({conv_id[:8]})",
|
||||
FileNotFoundError(f"No session file for id {conv_id}"),
|
||||
)
|
||||
|
||||
records: list[dict] = []
|
||||
bad_lines = 0
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
records.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
bad_lines += 1
|
||||
if bad_lines:
|
||||
logger.warning(
|
||||
"[claude-code] %s: skipped %d unparseable line(s)", path.name, bad_lines
|
||||
)
|
||||
|
||||
mtime = datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc)
|
||||
return {
|
||||
"id": conv_id,
|
||||
"_path": str(path),
|
||||
"_records": records,
|
||||
# Listing and normalized updated_at must match, or the cache
|
||||
# staleness comparison would re-export every session every run.
|
||||
"_mtime_iso": mtime.isoformat(),
|
||||
}
|
||||
|
||||
def normalize_conversation(self, raw: dict, loss_report: LossReport | None = None) -> dict:
|
||||
report = loss_report if loss_report is not None else LossReport()
|
||||
policy = getattr(self, "_hidden_content", None) or resolve_hidden_content_policy()
|
||||
conv_id = raw.get("id") or ""
|
||||
records: list[dict] = raw.get("_records") or []
|
||||
|
||||
title = _extract_title(records)
|
||||
project = _extract_project(records, raw.get("_path"))
|
||||
created_at = next(
|
||||
(r.get("timestamp") for r in records if r.get("timestamp")), ""
|
||||
)
|
||||
updated_at = raw.get("_mtime_iso") or next(
|
||||
(r.get("timestamp") for r in reversed(records) if r.get("timestamp")), ""
|
||||
)
|
||||
|
||||
messages = _extract_messages(records, conv_id, report, policy)
|
||||
for _ in messages:
|
||||
report.record_message()
|
||||
report.record_conversation()
|
||||
|
||||
return {
|
||||
"id": conv_id,
|
||||
"title": title,
|
||||
"provider": self.provider_name,
|
||||
"project": project,
|
||||
"created_at": created_at or "",
|
||||
"updated_at": updated_at or "",
|
||||
"message_count": len(messages),
|
||||
"messages": messages,
|
||||
}
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Scanning
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _scan(self) -> list[dict]:
|
||||
if not self._projects_dir.is_dir():
|
||||
logger.warning(
|
||||
"[claude-code] Projects directory %s does not exist", self._projects_dir
|
||||
)
|
||||
return []
|
||||
|
||||
convs: list[dict] = []
|
||||
for proj_dir in sorted(p for p in self._projects_dir.iterdir() if p.is_dir()):
|
||||
for session_file in sorted(proj_dir.glob("*.jsonl")):
|
||||
try:
|
||||
stat = session_file.stat()
|
||||
except OSError:
|
||||
continue
|
||||
if stat.st_size == 0:
|
||||
continue
|
||||
conv_id = session_file.stem
|
||||
self._path_map[conv_id] = session_file
|
||||
title, project, created = _read_session_meta(session_file)
|
||||
convs.append(
|
||||
{
|
||||
"id": conv_id,
|
||||
"title": title,
|
||||
"project": project,
|
||||
# The --project filter and dry-run table read the
|
||||
# listing dict, not the normalized conversation.
|
||||
"_project_name": project,
|
||||
"created_at": created,
|
||||
"updated_at": datetime.fromtimestamp(
|
||||
stat.st_mtime, tz=timezone.utc
|
||||
).isoformat(),
|
||||
"_path": str(session_file),
|
||||
"_project_dir": proj_dir.name,
|
||||
}
|
||||
)
|
||||
return convs
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Internal helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _read_session_meta(path: Path) -> tuple[str, str | None, str]:
|
||||
"""Light single-pass scan for listing metadata: (title, project, created_at).
|
||||
|
||||
Substring guards keep this cheap — only candidate lines are JSON-parsed.
|
||||
The full-fidelity extraction happens later in normalize_conversation.
|
||||
"""
|
||||
title = ""
|
||||
project: str | None = None
|
||||
created = ""
|
||||
try:
|
||||
with path.open(encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
if '"ai-title"' in line:
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if rec.get("type") == "ai-title" and rec.get("aiTitle"):
|
||||
title = str(rec["aiTitle"]) # last one wins
|
||||
continue
|
||||
if (not created or project is None) and (
|
||||
'"timestamp"' in line or '"cwd"' in line
|
||||
):
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if not created and rec.get("timestamp"):
|
||||
created = str(rec["timestamp"])
|
||||
if project is None and rec.get("cwd"):
|
||||
project = Path(rec["cwd"]).name or None
|
||||
except OSError as e:
|
||||
logger.warning("[claude-code] Could not read %s: %s", path, e)
|
||||
return title, project, created
|
||||
|
||||
|
||||
def _extract_title(records: list[dict]) -> str:
|
||||
"""Last ai-title record wins; fall back to the first real user prompt."""
|
||||
title = ""
|
||||
for rec in records:
|
||||
if rec.get("type") == "ai-title" and rec.get("aiTitle"):
|
||||
title = str(rec["aiTitle"])
|
||||
if title:
|
||||
return title
|
||||
|
||||
for rec in records:
|
||||
if rec.get("type") != "user" or rec.get("isMeta") or rec.get("isSidechain"):
|
||||
continue
|
||||
content = (rec.get("message") or {}).get("content")
|
||||
if isinstance(content, str):
|
||||
text = _strip_harness_noise(content)
|
||||
elif isinstance(content, list):
|
||||
text = " ".join(
|
||||
_strip_harness_noise(item.get("text", ""))
|
||||
for item in content
|
||||
if isinstance(item, dict) and item.get("type") == "text"
|
||||
).strip()
|
||||
else:
|
||||
text = ""
|
||||
if text:
|
||||
return text[:80]
|
||||
return "Untitled session"
|
||||
|
||||
|
||||
def _extract_project(records: list[dict], path: str | None) -> str | None:
|
||||
"""Project = basename of the session's working directory."""
|
||||
for rec in records:
|
||||
cwd = rec.get("cwd")
|
||||
if cwd:
|
||||
name = Path(cwd).name
|
||||
if name:
|
||||
return name
|
||||
# Fallback: the munged directory name (cannot be reliably de-munged
|
||||
# because '-' is both the path separator and a legal name character).
|
||||
if path:
|
||||
return Path(path).parent.name.lstrip("-") or None
|
||||
return None
|
||||
|
||||
|
||||
def _extract_messages(
|
||||
records: list[dict], conv_id: str, report: LossReport, policy: str
|
||||
) -> list[dict]:
|
||||
messages: list[dict] = []
|
||||
# Pending collapsed tool activity: name → call count, plus total bytes.
|
||||
pending_tools: Counter = Counter()
|
||||
pending_bytes = 0
|
||||
|
||||
def flush_pending() -> None:
|
||||
nonlocal pending_bytes
|
||||
if not pending_tools:
|
||||
return
|
||||
shown = ", ".join(
|
||||
f"{name} ×{count}" for name, count in pending_tools.most_common(_TOOL_NAMES_SHOWN)
|
||||
)
|
||||
if len(pending_tools) > _TOOL_NAMES_SHOWN:
|
||||
shown += ", …"
|
||||
calls = sum(pending_tools.values())
|
||||
block = make_collapsed_block(
|
||||
origin=f"{calls} calls: {shown}",
|
||||
content_type="tool_activity",
|
||||
size_bytes=pending_bytes,
|
||||
kind=COLLAPSED_KIND_TOOL_DUMP,
|
||||
)
|
||||
if messages and messages[-1]["role"] == "assistant":
|
||||
messages[-1]["blocks"].append(block)
|
||||
else:
|
||||
messages.append(
|
||||
{"role": "tool", "content_type": "text", "timestamp": None, "blocks": [block]}
|
||||
)
|
||||
pending_tools.clear()
|
||||
pending_bytes = 0
|
||||
|
||||
for rec in records:
|
||||
if rec.get("type") not in ("user", "assistant"):
|
||||
continue
|
||||
if rec.get("isSidechain") or rec.get("isMeta"):
|
||||
continue
|
||||
|
||||
msg = rec.get("message") or {}
|
||||
role = msg.get("role") or rec["type"]
|
||||
content = msg.get("content")
|
||||
blocks: list[dict] = []
|
||||
# This record's own tool traffic — merged into pending only after the
|
||||
# record's message is appended, so the placeholder lands on it (not on
|
||||
# the previous message).
|
||||
local_tools: Counter = Counter()
|
||||
local_bytes = 0
|
||||
|
||||
if isinstance(content, str):
|
||||
text = _strip_harness_noise(content)
|
||||
block = make_text_block(text)
|
||||
if block:
|
||||
blocks.append(block)
|
||||
elif isinstance(content, list):
|
||||
for item in content:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
item_type = item.get("type", "")
|
||||
if item_type == "text":
|
||||
block = make_text_block(_strip_harness_noise(item.get("text", "")))
|
||||
if block:
|
||||
blocks.append(block)
|
||||
elif item_type in ("thinking", "redacted_thinking"):
|
||||
if policy == HIDDEN_CONTENT_FULL:
|
||||
block = make_thinking_block(
|
||||
item.get("thinking") or item.get("text") or ""
|
||||
)
|
||||
if block:
|
||||
blocks.append(block)
|
||||
else:
|
||||
# Decision 2026-06-12: thinking is dropped without a
|
||||
# placeholder, but stays visible in the run summary.
|
||||
report.record_collapsed(
|
||||
"thinking", len(json.dumps(item, default=str))
|
||||
)
|
||||
elif item_type == "tool_use":
|
||||
if policy == HIDDEN_CONTENT_FULL:
|
||||
blocks.append(
|
||||
make_tool_use_block(
|
||||
item.get("name", ""), item.get("input"), item.get("id")
|
||||
)
|
||||
)
|
||||
else:
|
||||
name = item.get("name") or "tool"
|
||||
size = len(json.dumps(item, default=str))
|
||||
local_tools[name] += 1
|
||||
local_bytes += size
|
||||
report.record_collapsed(name, size)
|
||||
elif item_type == "tool_result":
|
||||
if policy == HIDDEN_CONTENT_FULL:
|
||||
blocks.append(
|
||||
make_tool_result_block(
|
||||
_stringify_tool_result(item.get("content")),
|
||||
is_error=bool(item.get("is_error")),
|
||||
)
|
||||
)
|
||||
else:
|
||||
size = len(json.dumps(item, default=str))
|
||||
local_bytes += size
|
||||
report.record_collapsed("tool_result", size)
|
||||
elif item_type == "image":
|
||||
blocks.append(
|
||||
make_image_placeholder(ref="embedded image", source="user_upload")
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"[claude-code] Unknown content block type %r in session %s",
|
||||
item_type,
|
||||
conv_id[:8],
|
||||
)
|
||||
report.record_unknown(f"claude-code.{item_type or '?'}")
|
||||
blocks.append(
|
||||
make_unknown_block(
|
||||
raw_type=f"claude-code.{item_type or '?'}",
|
||||
observed_keys=list(item.keys()),
|
||||
reason=UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
)
|
||||
)
|
||||
|
||||
if not blocks:
|
||||
# Tool-result-only records (and similar): traffic accumulates and
|
||||
# is attached to the message that initiated it.
|
||||
pending_tools.update(local_tools)
|
||||
pending_bytes += local_bytes
|
||||
continue
|
||||
|
||||
# Attach previous records' tool activity to the previous message
|
||||
# before starting a new one, so the placeholder lands between the
|
||||
# dialogue turns it actually occurred between.
|
||||
flush_pending()
|
||||
messages.append(
|
||||
{
|
||||
"role": role,
|
||||
"content_type": "text",
|
||||
"timestamp": rec.get("timestamp"),
|
||||
"blocks": blocks,
|
||||
}
|
||||
)
|
||||
pending_tools.update(local_tools)
|
||||
pending_bytes += local_bytes
|
||||
|
||||
flush_pending()
|
||||
return messages
|
||||
|
||||
|
||||
def _stringify_tool_result(content) -> str:
|
||||
"""tool_result content may be a string or a list of text blocks."""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts = []
|
||||
for item in content:
|
||||
if isinstance(item, dict) and item.get("type") == "text":
|
||||
parts.append(item.get("text", ""))
|
||||
else:
|
||||
parts.append(json.dumps(item, default=str))
|
||||
return "\n".join(parts)
|
||||
return json.dumps(content, default=str) if content is not None else ""
|
||||
+3
-3
@@ -50,7 +50,7 @@ def build_export_path(
|
||||
created_at: ISO8601 creation timestamp (used for year folder).
|
||||
filename: Already-generated filename from generate_filename().
|
||||
structure: OUTPUT_STRUCTURE value. One of:
|
||||
"provider/project/year" (default)
|
||||
"provider/project/year" (default) — project and year combined, e.g. no-project.2025/
|
||||
"provider/project"
|
||||
"provider/year"
|
||||
|
||||
@@ -64,14 +64,14 @@ def build_export_path(
|
||||
parts: list[str] = [provider]
|
||||
|
||||
if structure == "provider/project/year":
|
||||
parts += [project_slug, year]
|
||||
parts += [f"{project_slug}.{year}"]
|
||||
elif structure == "provider/project":
|
||||
parts += [project_slug]
|
||||
elif structure == "provider/year":
|
||||
parts += [year]
|
||||
else:
|
||||
# Unknown structure — fall back to default
|
||||
parts += [project_slug, year]
|
||||
parts += [f"{project_slug}.{year}"]
|
||||
|
||||
return base_dir.joinpath(*parts) / filename
|
||||
|
||||
|
||||
+148
-9
@@ -8,12 +8,30 @@
|
||||
"node-root": {
|
||||
"id": "node-root",
|
||||
"parent": null,
|
||||
"children": ["node-1"],
|
||||
"children": ["node-uec"],
|
||||
"message": null
|
||||
},
|
||||
"node-uec": {
|
||||
"id": "node-uec",
|
||||
"parent": "node-root",
|
||||
"children": ["node-1"],
|
||||
"message": {
|
||||
"id": "node-uec",
|
||||
"author": {"role": "user"},
|
||||
"create_time": null,
|
||||
"content": {
|
||||
"content_type": "user_editable_context",
|
||||
"user_profile": "Preferred name: Jesse",
|
||||
"user_instructions": "The user provided the additional info about how they would like you to respond:\n```Always cite sources.```"
|
||||
},
|
||||
"metadata": {
|
||||
"is_visually_hidden_from_conversation": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node-1": {
|
||||
"id": "node-1",
|
||||
"parent": "node-root",
|
||||
"parent": "node-uec",
|
||||
"children": ["node-2"],
|
||||
"message": {
|
||||
"id": "node-1",
|
||||
@@ -28,7 +46,7 @@
|
||||
"node-2": {
|
||||
"id": "node-2",
|
||||
"parent": "node-1",
|
||||
"children": ["node-3"],
|
||||
"children": ["node-mm-user"],
|
||||
"message": {
|
||||
"id": "node-2",
|
||||
"author": {"role": "assistant"},
|
||||
@@ -39,18 +57,139 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"node-3": {
|
||||
"id": "node-3",
|
||||
"node-mm-user": {
|
||||
"id": "node-mm-user",
|
||||
"parent": "node-2",
|
||||
"children": [],
|
||||
"children": ["node-mm-assistant"],
|
||||
"message": {
|
||||
"id": "node-3",
|
||||
"id": "node-mm-user",
|
||||
"author": {"role": "user"},
|
||||
"create_time": 1704067300.0,
|
||||
"content": {
|
||||
"content_type": "image_asset_pointer",
|
||||
"parts": [{"content_type": "image_asset_pointer", "asset_pointer": "file://some-image"}]
|
||||
"content_type": "multimodal_text",
|
||||
"parts": [
|
||||
{"content_type": "audio_transcription", "text": "What is the capital of France?", "direction": "in", "decoding_id": null},
|
||||
{"content_type": "real_time_user_audio_video_asset_pointer", "frames_asset_pointers": [], "video_container_asset_pointer": null, "audio_asset_pointer": {"content_type": "audio_asset_pointer", "asset_pointer": "sediment://file_user001", "size_bytes": 50000, "format": "wav", "metadata": {"start": 0.0, "end": 2.5}}, "audio_start_timestamp": 1.0}
|
||||
]
|
||||
},
|
||||
"metadata": {"voice_mode_message": true}
|
||||
}
|
||||
},
|
||||
"node-mm-assistant": {
|
||||
"id": "node-mm-assistant",
|
||||
"parent": "node-mm-user",
|
||||
"children": ["node-mm-user-rev"],
|
||||
"message": {
|
||||
"id": "node-mm-assistant",
|
||||
"author": {"role": "assistant"},
|
||||
"create_time": 1704067305.0,
|
||||
"content": {
|
||||
"content_type": "multimodal_text",
|
||||
"parts": [
|
||||
{"content_type": "audio_transcription", "text": "The capital of France is Paris.", "direction": "out", "decoding_id": null},
|
||||
{"content_type": "audio_asset_pointer", "asset_pointer": "sediment://file_assistant001", "size_bytes": 80000, "format": "wav", "metadata": {"start": 0.0, "end": 3.2}}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"node-mm-user-rev": {
|
||||
"id": "node-mm-user-rev",
|
||||
"parent": "node-mm-assistant",
|
||||
"children": ["node-image-only"],
|
||||
"message": {
|
||||
"id": "node-mm-user-rev",
|
||||
"author": {"role": "user"},
|
||||
"create_time": 1704067400.0,
|
||||
"content": {
|
||||
"content_type": "multimodal_text",
|
||||
"parts": [
|
||||
{"content_type": "real_time_user_audio_video_asset_pointer", "frames_asset_pointers": [], "video_container_asset_pointer": null, "audio_asset_pointer": {"content_type": "audio_asset_pointer", "asset_pointer": "sediment://file_user002", "size_bytes": 30000, "format": "wav", "metadata": {"start": 0.0, "end": 1.5}}, "audio_start_timestamp": 5.0},
|
||||
{"content_type": "audio_transcription", "text": "Tell me more please.", "direction": "in", "decoding_id": null}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"node-image-only": {
|
||||
"id": "node-image-only",
|
||||
"parent": "node-mm-user-rev",
|
||||
"children": ["node-exec-output"],
|
||||
"message": {
|
||||
"id": "node-image-only",
|
||||
"author": {"role": "user"},
|
||||
"create_time": 1704067500.0,
|
||||
"content": {
|
||||
"content_type": "multimodal_text",
|
||||
"parts": [
|
||||
{"content_type": "image_asset_pointer", "asset_pointer": "file-service://image001"}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"node-exec-output": {
|
||||
"id": "node-exec-output",
|
||||
"parent": "node-image-only",
|
||||
"children": ["node-exec-output-empty"],
|
||||
"message": {
|
||||
"id": "node-exec-output",
|
||||
"author": {"role": "tool", "name": "container.exec", "metadata": {}},
|
||||
"create_time": 1704067600.0,
|
||||
"content": {
|
||||
"content_type": "execution_output",
|
||||
"text": "Hello from container.exec\nLine 2 of output"
|
||||
},
|
||||
"metadata": {
|
||||
"aggregate_result": {"status": "success", "messages": []},
|
||||
"reasoning_title": "Reading skill documentation"
|
||||
}
|
||||
}
|
||||
},
|
||||
"node-exec-output-empty": {
|
||||
"id": "node-exec-output-empty",
|
||||
"parent": "node-exec-output",
|
||||
"children": ["node-system-error"],
|
||||
"message": {
|
||||
"id": "node-exec-output-empty",
|
||||
"author": {"role": "tool", "name": "python", "metadata": {}},
|
||||
"create_time": 1704067610.0,
|
||||
"content": {
|
||||
"content_type": "execution_output",
|
||||
"text": ""
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
},
|
||||
"node-system-error": {
|
||||
"id": "node-system-error",
|
||||
"parent": "node-exec-output-empty",
|
||||
"children": ["node-tether-spinner"],
|
||||
"message": {
|
||||
"id": "node-system-error",
|
||||
"author": {"role": "tool", "name": "web", "metadata": {}},
|
||||
"create_time": 1704067620.0,
|
||||
"content": {
|
||||
"content_type": "system_error",
|
||||
"name": "tool_error",
|
||||
"text": "Error: Error from browse service: Error calling browse service: 503"
|
||||
},
|
||||
"metadata": {}
|
||||
}
|
||||
},
|
||||
"node-tether-spinner": {
|
||||
"id": "node-tether-spinner",
|
||||
"parent": "node-system-error",
|
||||
"children": [],
|
||||
"message": {
|
||||
"id": "node-tether-spinner",
|
||||
"author": {"role": "tool", "name": "file_search", "metadata": {}},
|
||||
"create_time": 1704067630.0,
|
||||
"content": {
|
||||
"content_type": "tether_browsing_display",
|
||||
"result": "",
|
||||
"summary": "",
|
||||
"assets": null,
|
||||
"tether_id": null
|
||||
},
|
||||
"metadata": {"command": "spinner", "status": "running"}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+9
@@ -30,6 +30,15 @@
|
||||
"sender": "human",
|
||||
"created_at": "2024-06-10T14:45:00.000Z",
|
||||
"content": "Thank you, that helped!"
|
||||
},
|
||||
{
|
||||
"uuid": "msg-004",
|
||||
"sender": "human",
|
||||
"created_at": "2024-06-10T14:50:00.000Z",
|
||||
"content": [
|
||||
{"type": "text", "text": "What about this image?"},
|
||||
{"type": "image", "source": {"file_uuid": "claude-image-uuid-1", "media_type": "image/png"}}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
"""Unit tests for browser cookie extraction (browser_cookie3 mocked)."""
|
||||
|
||||
import sys
|
||||
import types
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from src.browser_tokens import (
|
||||
BrowserTokenError,
|
||||
extract_chatgpt_tokens,
|
||||
extract_claude_session,
|
||||
)
|
||||
|
||||
|
||||
def _fake_bc3(monkeypatch, cookies_by_domain, raises=None):
|
||||
"""Install a stub browser_cookie3 module whose ``brave`` loader returns
|
||||
SimpleNamespace cookies for the requested domain (or raises)."""
|
||||
module = types.ModuleType("browser_cookie3")
|
||||
|
||||
def brave(domain_name=""):
|
||||
if raises is not None:
|
||||
raise raises
|
||||
return [
|
||||
SimpleNamespace(name=name, value=value)
|
||||
for name, value in cookies_by_domain.get(domain_name, {}).items()
|
||||
]
|
||||
|
||||
module.brave = brave
|
||||
monkeypatch.setitem(sys.modules, "browser_cookie3", module)
|
||||
return module
|
||||
|
||||
|
||||
class TestExtractChatGPT:
|
||||
def test_chunked_cookies(self, monkeypatch):
|
||||
_fake_bc3(monkeypatch, {
|
||||
"chatgpt.com": {
|
||||
"__Secure-next-auth.session-token.0": "eyJpart0",
|
||||
"__Secure-next-auth.session-token.1": "part1rest",
|
||||
},
|
||||
})
|
||||
assert extract_chatgpt_tokens("brave") == ("eyJpart0", "part1rest")
|
||||
|
||||
def test_single_cookie_fallback(self, monkeypatch):
|
||||
_fake_bc3(monkeypatch, {
|
||||
"chatgpt.com": {"__Secure-next-auth.session-token": "eyJsingle"},
|
||||
})
|
||||
assert extract_chatgpt_tokens("brave") == ("eyJsingle", None)
|
||||
|
||||
def test_missing_cookie_raises_with_login_hint(self, monkeypatch):
|
||||
_fake_bc3(monkeypatch, {"chatgpt.com": {"unrelated": "x"}})
|
||||
with pytest.raises(BrowserTokenError, match="log in"):
|
||||
extract_chatgpt_tokens("brave")
|
||||
|
||||
def test_loader_failure_wrapped(self, monkeypatch):
|
||||
_fake_bc3(monkeypatch, {}, raises=RuntimeError("database is locked"))
|
||||
with pytest.raises(BrowserTokenError, match="Close the browser"):
|
||||
extract_chatgpt_tokens("brave")
|
||||
|
||||
def test_unsupported_browser_rejected(self):
|
||||
with pytest.raises(BrowserTokenError, match="Unsupported browser"):
|
||||
extract_chatgpt_tokens("netscape")
|
||||
|
||||
|
||||
class TestExtractClaude:
|
||||
def test_session_key(self, monkeypatch):
|
||||
_fake_bc3(monkeypatch, {"claude.ai": {"sessionKey": "sk-ant-sid01-xyz"}})
|
||||
assert extract_claude_session("brave") == "sk-ant-sid01-xyz"
|
||||
|
||||
def test_missing_raises(self, monkeypatch):
|
||||
_fake_bc3(monkeypatch, {"claude.ai": {}})
|
||||
with pytest.raises(BrowserTokenError, match="log in"):
|
||||
extract_claude_session("brave")
|
||||
|
||||
|
||||
class TestAuthFromBrowserCLI:
|
||||
def test_writes_env_after_validation(self, monkeypatch, tmp_path):
|
||||
"""--from-browser extracts, validates live (mocked), and writes .env."""
|
||||
from click.testing import CliRunner
|
||||
from src.main import cli
|
||||
from src.cache import Cache
|
||||
|
||||
cache = Cache(tmp_path / "cache")
|
||||
cache.acknowledge_tos()
|
||||
|
||||
_fake_bc3(monkeypatch, {
|
||||
"chatgpt.com": {
|
||||
"__Secure-next-auth.session-token.0": "eyJtok0",
|
||||
"__Secure-next-auth.session-token.1": "tok1",
|
||||
},
|
||||
"claude.ai": {"sessionKey": "sk-ant-sid01-abc"},
|
||||
})
|
||||
|
||||
# Stub out the live validation calls.
|
||||
import src.providers.chatgpt as chatgpt_mod
|
||||
import src.providers.claude as claude_mod
|
||||
monkeypatch.setattr(
|
||||
chatgpt_mod.ChatGPTProvider, "_fetch_access_token", lambda self: "at"
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
claude_mod.ClaudeProvider,
|
||||
"list_conversations",
|
||||
lambda self, offset=0, limit=100: [],
|
||||
)
|
||||
|
||||
runner = CliRunner(mix_stderr=True)
|
||||
with runner.isolated_filesystem(temp_dir=tmp_path):
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["--no-log-file", "auth", "--from-browser"],
|
||||
env={"CACHE_DIR": str(tmp_path / "cache")},
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
from pathlib import Path
|
||||
env_text = Path(".env").read_text()
|
||||
|
||||
assert "CHATGPT_SESSION_TOKEN=eyJtok0" in env_text
|
||||
assert "CHATGPT_SESSION_TOKEN_1=tok1" in env_text
|
||||
assert "CLAUDE_SESSION_KEY=sk-ant-sid01-abc" in env_text
|
||||
assert "2 provider(s) configured" in result.output
|
||||
|
||||
def test_validation_failure_leaves_env_untouched(self, monkeypatch, tmp_path):
|
||||
from click.testing import CliRunner
|
||||
from src.main import cli
|
||||
from src.cache import Cache
|
||||
from src.providers.base import ProviderError
|
||||
|
||||
cache = Cache(tmp_path / "cache")
|
||||
cache.acknowledge_tos()
|
||||
|
||||
_fake_bc3(monkeypatch, {
|
||||
"chatgpt.com": {"__Secure-next-auth.session-token.0": "eyJstale"},
|
||||
"claude.ai": {},
|
||||
})
|
||||
|
||||
import src.providers.chatgpt as chatgpt_mod
|
||||
|
||||
def _fail(self):
|
||||
raise ProviderError("chatgpt", "auth", RuntimeError("401"))
|
||||
|
||||
monkeypatch.setattr(chatgpt_mod.ChatGPTProvider, "_fetch_access_token", _fail)
|
||||
|
||||
runner = CliRunner(mix_stderr=True)
|
||||
with runner.isolated_filesystem(temp_dir=tmp_path):
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["--no-log-file", "auth", "--from-browser", "brave"],
|
||||
env={"CACHE_DIR": str(tmp_path / "cache")},
|
||||
)
|
||||
import pathlib
|
||||
env_exists = pathlib.Path(".env").exists()
|
||||
|
||||
assert result.exit_code == 1
|
||||
assert "failed live validation" in result.output
|
||||
assert not env_exists
|
||||
@@ -0,0 +1,230 @@
|
||||
"""Unit tests for the Claude Code session provider."""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from src.blocks import (
|
||||
BLOCK_TYPE_COLLAPSED,
|
||||
BLOCK_TYPE_TEXT,
|
||||
BLOCK_TYPE_THINKING,
|
||||
BLOCK_TYPE_TOOL_RESULT,
|
||||
BLOCK_TYPE_TOOL_USE,
|
||||
render_blocks_to_markdown,
|
||||
)
|
||||
from src.loss_report import LossReport
|
||||
from src.providers.claude_code import ClaudeCodeProvider
|
||||
|
||||
|
||||
def _write_session(tmp_path, records, project_dir="-home-jesse-myproj", name="abc-123"):
|
||||
proj = tmp_path / project_dir
|
||||
proj.mkdir(parents=True, exist_ok=True)
|
||||
f = proj / f"{name}.jsonl"
|
||||
f.write_text("\n".join(json.dumps(r) for r in records), encoding="utf-8")
|
||||
return f
|
||||
|
||||
|
||||
def _records():
|
||||
"""A representative session: title, harness noise, dialogue, tool traffic."""
|
||||
return [
|
||||
{"type": "ai-title", "aiTitle": "First title", "sessionId": "abc-123"},
|
||||
{
|
||||
"type": "user",
|
||||
"cwd": "/home/jesse/myproj",
|
||||
"timestamp": "2026-05-01T10:00:00.000Z",
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": (
|
||||
"<local-command-caveat>Caveat: ignore</local-command-caveat>"
|
||||
"<command-name>/model</command-name>"
|
||||
"<local-command-stdout>Set model</local-command-stdout>"
|
||||
"Please review my project."
|
||||
),
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"timestamp": "2026-05-01T10:00:05.000Z",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{"type": "thinking", "thinking": "Let me think about this."},
|
||||
{"type": "text", "text": "I'll review it now."},
|
||||
{"type": "tool_use", "id": "t1", "name": "Read", "input": {"file_path": "/x"}},
|
||||
{"type": "tool_use", "id": "t2", "name": "Read", "input": {"file_path": "/y"}},
|
||||
{"type": "tool_use", "id": "t3", "name": "Bash", "input": {"command": "ls"}},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"timestamp": "2026-05-01T10:00:08.000Z",
|
||||
"message": {
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "tool_result", "tool_use_id": "t1", "content": "file contents " * 50},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "assistant",
|
||||
"timestamp": "2026-05-01T10:00:20.000Z",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": [{"type": "text", "text": "Here is my review: all good."}],
|
||||
},
|
||||
},
|
||||
# Harness records — skipped
|
||||
{"type": "file-history-snapshot", "snapshot": {"big": "blob"}},
|
||||
{"type": "permission-mode", "mode": "plan"},
|
||||
# Subagent and meta records — skipped
|
||||
{
|
||||
"type": "assistant",
|
||||
"isSidechain": True,
|
||||
"message": {"role": "assistant", "content": [{"type": "text", "text": "subagent noise"}]},
|
||||
},
|
||||
{
|
||||
"type": "user",
|
||||
"isMeta": True,
|
||||
"message": {"role": "user", "content": "meta noise"},
|
||||
},
|
||||
# Later title wins
|
||||
{"type": "ai-title", "aiTitle": "Review my project", "sessionId": "abc-123"},
|
||||
]
|
||||
|
||||
|
||||
class TestClaudeCodeProvider:
|
||||
def _provider(self, tmp_path, policy="placeholder"):
|
||||
return ClaudeCodeProvider(projects_dir=tmp_path, hidden_content=policy)
|
||||
|
||||
def test_listing_metadata(self, tmp_path):
|
||||
_write_session(tmp_path, _records())
|
||||
p = self._provider(tmp_path)
|
||||
convs = p.fetch_all_conversations()
|
||||
assert len(convs) == 1
|
||||
c = convs[0]
|
||||
assert c["id"] == "abc-123"
|
||||
assert c["title"] == "Review my project" # last ai-title wins
|
||||
assert c["_project_name"] == "myproj" # from cwd basename
|
||||
assert c["created_at"] == "2026-05-01T10:00:00.000Z"
|
||||
assert c["updated_at"] # file mtime
|
||||
|
||||
def test_empty_files_skipped(self, tmp_path):
|
||||
proj = tmp_path / "-home-x"
|
||||
proj.mkdir()
|
||||
(proj / "empty.jsonl").write_text("")
|
||||
p = self._provider(tmp_path)
|
||||
assert p.fetch_all_conversations() == []
|
||||
|
||||
def test_normalize_prose_only_default(self, tmp_path):
|
||||
_write_session(tmp_path, _records())
|
||||
p = self._provider(tmp_path)
|
||||
p.fetch_all_conversations()
|
||||
raw = p.get_conversation("abc-123")
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
assert result["provider"] == "claude-code"
|
||||
assert result["title"] == "Review my project"
|
||||
assert result["project"] == "myproj"
|
||||
|
||||
roles = [m["role"] for m in result["messages"]]
|
||||
assert roles == ["user", "assistant", "assistant"]
|
||||
|
||||
# Harness tags stripped, dialogue kept
|
||||
user_text = result["messages"][0]["blocks"][0]["text"]
|
||||
assert user_text == "Please review my project."
|
||||
assert "Caveat" not in user_text
|
||||
|
||||
# Tool activity collapsed into one grouped placeholder on the
|
||||
# assistant message that ran the tools (includes the tool_result
|
||||
# bytes from the following user record).
|
||||
first_assistant = result["messages"][1]
|
||||
types = [b["type"] for b in first_assistant["blocks"]]
|
||||
assert types == [BLOCK_TYPE_TEXT, BLOCK_TYPE_COLLAPSED]
|
||||
collapsed = first_assistant["blocks"][1]
|
||||
assert "3 calls" in collapsed["origin"]
|
||||
assert "Read ×2" in collapsed["origin"]
|
||||
assert "Bash ×1" in collapsed["origin"]
|
||||
assert collapsed["size_bytes"] > 500
|
||||
|
||||
# Thinking dropped without placeholder; sidechain/meta absent
|
||||
all_blocks = [b for m in result["messages"] for b in m["blocks"]]
|
||||
assert not any(b["type"] == BLOCK_TYPE_THINKING for b in all_blocks)
|
||||
assert not any(
|
||||
"subagent noise" in (b.get("text") or "") or "meta noise" in (b.get("text") or "")
|
||||
for b in all_blocks
|
||||
)
|
||||
|
||||
def test_collapsed_counted_in_loss_report(self, tmp_path):
|
||||
_write_session(tmp_path, _records())
|
||||
p = self._provider(tmp_path)
|
||||
p.fetch_all_conversations()
|
||||
report = LossReport()
|
||||
p.normalize_conversation(p.get_conversation("abc-123"), report)
|
||||
assert report.collapsed["Read"] == 2
|
||||
assert report.collapsed["Bash"] == 1
|
||||
assert report.collapsed["tool_result"] == 1
|
||||
assert report.collapsed["thinking"] == 1
|
||||
assert report.collapsed_bytes > 500
|
||||
|
||||
def test_full_policy_keeps_everything(self, tmp_path):
|
||||
_write_session(tmp_path, _records())
|
||||
p = self._provider(tmp_path, policy="full")
|
||||
p.fetch_all_conversations()
|
||||
result = p.normalize_conversation(p.get_conversation("abc-123"))
|
||||
all_blocks = [b for m in result["messages"] for b in m["blocks"]]
|
||||
types = {b["type"] for b in all_blocks}
|
||||
assert BLOCK_TYPE_THINKING in types
|
||||
assert BLOCK_TYPE_TOOL_USE in types
|
||||
assert BLOCK_TYPE_TOOL_RESULT in types
|
||||
assert BLOCK_TYPE_COLLAPSED not in types
|
||||
|
||||
def test_updated_at_consistent_with_listing(self, tmp_path):
|
||||
"""Listing and normalized updated_at must match or the cache would
|
||||
consider every session stale on every run (mtime vs last timestamp)."""
|
||||
_write_session(tmp_path, _records())
|
||||
p = self._provider(tmp_path)
|
||||
listing = p.fetch_all_conversations()[0]
|
||||
normalized = p.normalize_conversation(p.get_conversation("abc-123"))
|
||||
assert normalized["updated_at"] == listing["updated_at"]
|
||||
|
||||
def test_title_falls_back_to_first_prompt(self, tmp_path):
|
||||
records = [r for r in _records() if r.get("type") != "ai-title"]
|
||||
_write_session(tmp_path, records)
|
||||
p = self._provider(tmp_path)
|
||||
p.fetch_all_conversations()
|
||||
result = p.normalize_conversation(p.get_conversation("abc-123"))
|
||||
assert result["title"] == "Please review my project."
|
||||
|
||||
def test_unknown_block_type_surfaces(self, tmp_path):
|
||||
records = [
|
||||
{
|
||||
"type": "assistant",
|
||||
"timestamp": "2026-05-01T10:00:00.000Z",
|
||||
"cwd": "/home/jesse/myproj",
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": [
|
||||
{"type": "text", "text": "hello"},
|
||||
{"type": "future_block_xyz", "data": 1},
|
||||
],
|
||||
},
|
||||
},
|
||||
]
|
||||
_write_session(tmp_path, records)
|
||||
p = self._provider(tmp_path)
|
||||
p.fetch_all_conversations()
|
||||
report = LossReport()
|
||||
result = p.normalize_conversation(p.get_conversation("abc-123"), report)
|
||||
assert report.unknown_blocks["claude-code.future_block_xyz"] == 1
|
||||
rendered = render_blocks_to_markdown(result["messages"][0]["blocks"])
|
||||
assert "Unsupported content" in rendered
|
||||
|
||||
def test_collapsed_placeholder_renders_grouped_line(self, tmp_path):
|
||||
_write_session(tmp_path, _records())
|
||||
p = self._provider(tmp_path)
|
||||
p.fetch_all_conversations()
|
||||
result = p.normalize_conversation(p.get_conversation("abc-123"))
|
||||
rendered = render_blocks_to_markdown(result["messages"][1]["blocks"])
|
||||
assert "> 🔧 **Tool output** — `3 calls: Read ×2, Bash ×1`" in rendered
|
||||
assert "omitted (EXPORTER_HIDDEN_CONTENT=full to keep)" in rendered
|
||||
@@ -124,6 +124,154 @@ class TestExportSinceValidation:
|
||||
"CHATGPT_SESSION_TOKEN": "eyJtesttoken",
|
||||
"CACHE_DIR": str(tmp_path),
|
||||
"EXPORT_DIR": str(tmp_path / "exports"),
|
||||
# This test hits the real (failing) auth endpoint with retries;
|
||||
# don't add politeness pacing on top of the backoff sleeps.
|
||||
"REQUEST_DELAY": "0",
|
||||
},
|
||||
)
|
||||
assert "Invalid --since date" not in result.output
|
||||
|
||||
def test_max_conversations_zero_rejected(self, tmp_path):
|
||||
"""--max-conversations uses IntRange(min=1); 0 must be rejected by click."""
|
||||
self._pre_populated_cache(tmp_path)
|
||||
|
||||
runner = CliRunner(mix_stderr=True)
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["--no-log-file", "export", "--max-conversations", "0"],
|
||||
env={
|
||||
"CHATGPT_SESSION_TOKEN": "eyJtesttoken",
|
||||
"CACHE_DIR": str(tmp_path),
|
||||
"EXPORT_DIR": str(tmp_path / "exports"),
|
||||
},
|
||||
)
|
||||
assert result.exit_code == 2
|
||||
assert "max-conversations" in result.output
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LossReport summary
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestLossReportSummary:
|
||||
"""The LossReport's format_summary() pinned format covers zero, top-5, and overflow cases."""
|
||||
|
||||
def test_zero_summary_uses_none_sentinel(self):
|
||||
from src.loss_report import LossReport
|
||||
|
||||
report = LossReport()
|
||||
out = report.format_summary()
|
||||
assert "[export] Run summary:" in out
|
||||
assert "conversations: 0" in out
|
||||
assert "messages rendered: 0" in out
|
||||
# All three "(none)" sentinels present — never empty parens
|
||||
# (unknown blocks, extraction failures, collapsed by policy)
|
||||
assert out.count("(none)") == 3
|
||||
|
||||
def test_top_5_breakdown(self):
|
||||
from src.loss_report import LossReport
|
||||
|
||||
report = LossReport()
|
||||
for raw_type in ("a", "b", "c", "d", "e", "f", "g"):
|
||||
report.record_unknown(raw_type)
|
||||
if raw_type == "a":
|
||||
# Make 'a' the most common
|
||||
for _ in range(4):
|
||||
report.record_unknown("a")
|
||||
out = report.format_summary()
|
||||
# Top entry shown
|
||||
assert "a=5" in out
|
||||
# Overflow line present (7 types, top 5 + 2 more)
|
||||
assert "+ 2 more types" in out
|
||||
|
||||
def test_messages_and_conversations_recorded(self):
|
||||
from src.loss_report import LossReport
|
||||
|
||||
report = LossReport()
|
||||
report.record_conversation()
|
||||
report.record_message()
|
||||
report.record_message()
|
||||
out = report.format_summary()
|
||||
assert "conversations: 1" in out
|
||||
assert "messages rendered: 2" in out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# prune command
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPrune:
|
||||
"""prune deletes export files not referenced by the manifest."""
|
||||
|
||||
def _setup(self, tmp_path):
|
||||
"""Cache with one referenced file; one stale file + empty-dir candidate."""
|
||||
cache = Cache(tmp_path / "cache")
|
||||
cache.acknowledge_tos()
|
||||
export_dir = tmp_path / "exports"
|
||||
keep = export_dir / "chatgpt" / "proj.2026" / "keep.md"
|
||||
keep.parent.mkdir(parents=True)
|
||||
keep.write_text("kept")
|
||||
stale = export_dir / "chatgpt" / "proj" / "2026" / "old-layout.md"
|
||||
stale.parent.mkdir(parents=True)
|
||||
stale.write_text("stale")
|
||||
cache.mark_exported("chatgpt", "conv-1", {"file_path": str(keep)})
|
||||
return export_dir, keep, stale
|
||||
|
||||
def _invoke(self, tmp_path, *args):
|
||||
runner = CliRunner(mix_stderr=True)
|
||||
return runner.invoke(
|
||||
cli,
|
||||
["--no-log-file", "prune", *args],
|
||||
env={
|
||||
"CACHE_DIR": str(tmp_path / "cache"),
|
||||
"EXPORT_DIR": str(tmp_path / "exports"),
|
||||
},
|
||||
)
|
||||
|
||||
def test_dry_run_lists_but_keeps_files(self, tmp_path):
|
||||
export_dir, keep, stale = self._setup(tmp_path)
|
||||
result = self._invoke(tmp_path, "--dry-run")
|
||||
assert result.exit_code == 0
|
||||
assert "old-layout.md" in result.output
|
||||
assert "Dry run" in result.output
|
||||
assert stale.exists() and keep.exists()
|
||||
|
||||
def test_yes_deletes_stale_keeps_referenced_sweeps_dirs(self, tmp_path):
|
||||
export_dir, keep, stale = self._setup(tmp_path)
|
||||
result = self._invoke(tmp_path, "--yes")
|
||||
assert result.exit_code == 0
|
||||
assert not stale.exists()
|
||||
assert keep.exists()
|
||||
# Old-layout dirs are now empty and swept
|
||||
assert not (export_dir / "chatgpt" / "proj").exists()
|
||||
|
||||
def test_refuses_with_empty_manifest(self, tmp_path):
|
||||
"""Footgun guard: after cache --clear, prune must not wipe the archive."""
|
||||
cache = Cache(tmp_path / "cache")
|
||||
cache.acknowledge_tos()
|
||||
export_dir = tmp_path / "exports"
|
||||
f = export_dir / "chatgpt" / "a.md"
|
||||
f.parent.mkdir(parents=True)
|
||||
f.write_text("data")
|
||||
result = self._invoke(tmp_path, "--yes")
|
||||
assert result.exit_code == 1
|
||||
assert "Refusing to prune" in result.output
|
||||
assert f.exists()
|
||||
|
||||
def test_aborts_without_confirmation(self, tmp_path):
|
||||
export_dir, keep, stale = self._setup(tmp_path)
|
||||
runner = CliRunner(mix_stderr=True)
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["--no-log-file", "prune"],
|
||||
input="n\n",
|
||||
env={
|
||||
"CACHE_DIR": str(tmp_path / "cache"),
|
||||
"EXPORT_DIR": str(tmp_path / "exports"),
|
||||
},
|
||||
)
|
||||
assert result.exit_code == 0
|
||||
assert "Aborted" in result.output
|
||||
assert stale.exists()
|
||||
|
||||
@@ -54,3 +54,65 @@ class TestValidateChatGPTToken:
|
||||
result = _validate_chatgpt_token("notajwttoken")
|
||||
assert any("does not look like a JWT" in r.message for r in caplog.records)
|
||||
assert result is None
|
||||
|
||||
|
||||
class TestSessionLimiterConfig:
|
||||
"""MAX_CONVERSATIONS_PER_RUN and REQUEST_DELAY parsing in load_config."""
|
||||
|
||||
def _load(self, monkeypatch, tmp_path, **env):
|
||||
from src.config import load_config
|
||||
monkeypatch.setenv("EXPORT_DIR", str(tmp_path / "exports"))
|
||||
monkeypatch.setenv("CACHE_DIR", str(tmp_path / "cache"))
|
||||
for key in (
|
||||
"MAX_CONVERSATIONS_PER_RUN",
|
||||
"REQUEST_DELAY",
|
||||
"EXPORTER_HIDDEN_CONTENT",
|
||||
"EXPORTER_DOWNLOAD_MEDIA",
|
||||
):
|
||||
monkeypatch.delenv(key, raising=False)
|
||||
for key, value in env.items():
|
||||
monkeypatch.setenv(key, value)
|
||||
return load_config()
|
||||
|
||||
def test_defaults(self, monkeypatch, tmp_path):
|
||||
cfg = self._load(monkeypatch, tmp_path)
|
||||
assert cfg.max_conversations is None
|
||||
assert cfg.request_delay == 1.0
|
||||
assert cfg.hidden_content == "placeholder"
|
||||
assert cfg.download_media == "images"
|
||||
|
||||
def test_download_media_valid(self, monkeypatch, tmp_path):
|
||||
cfg = self._load(monkeypatch, tmp_path, EXPORTER_DOWNLOAD_MEDIA="all")
|
||||
assert cfg.download_media == "all"
|
||||
|
||||
def test_download_media_invalid_raises(self, monkeypatch, tmp_path):
|
||||
from src.config import ConfigError
|
||||
with pytest.raises(ConfigError, match="EXPORTER_DOWNLOAD_MEDIA"):
|
||||
self._load(monkeypatch, tmp_path, EXPORTER_DOWNLOAD_MEDIA="sometimes")
|
||||
|
||||
def test_valid_values(self, monkeypatch, tmp_path):
|
||||
cfg = self._load(
|
||||
monkeypatch, tmp_path,
|
||||
MAX_CONVERSATIONS_PER_RUN="25", REQUEST_DELAY="0.5",
|
||||
)
|
||||
assert cfg.max_conversations == 25
|
||||
assert cfg.request_delay == 0.5
|
||||
|
||||
def test_zero_delay_allowed(self, monkeypatch, tmp_path):
|
||||
cfg = self._load(monkeypatch, tmp_path, REQUEST_DELAY="0")
|
||||
assert cfg.request_delay == 0.0
|
||||
|
||||
def test_non_integer_cap_raises(self, monkeypatch, tmp_path):
|
||||
from src.config import ConfigError
|
||||
with pytest.raises(ConfigError, match="MAX_CONVERSATIONS_PER_RUN"):
|
||||
self._load(monkeypatch, tmp_path, MAX_CONVERSATIONS_PER_RUN="lots")
|
||||
|
||||
def test_zero_cap_raises(self, monkeypatch, tmp_path):
|
||||
from src.config import ConfigError
|
||||
with pytest.raises(ConfigError, match="at least 1"):
|
||||
self._load(monkeypatch, tmp_path, MAX_CONVERSATIONS_PER_RUN="0")
|
||||
|
||||
def test_negative_delay_raises(self, monkeypatch, tmp_path):
|
||||
from src.config import ConfigError
|
||||
with pytest.raises(ConfigError, match="REQUEST_DELAY"):
|
||||
self._load(monkeypatch, tmp_path, REQUEST_DELAY="-1")
|
||||
|
||||
+287
-2
@@ -1,4 +1,4 @@
|
||||
"""Unit tests for src/exporters/."""
|
||||
"""Unit tests for src/exporters/ and src/blocks.py."""
|
||||
|
||||
import json
|
||||
import os
|
||||
@@ -7,6 +7,23 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from src.blocks import (
|
||||
BLOCK_TYPE_TEXT,
|
||||
UNKNOWN_REASON_EXTRACTION_FAILED,
|
||||
UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
_blockquote_prefix,
|
||||
_safe_fence,
|
||||
make_code_block,
|
||||
make_file_placeholder,
|
||||
make_hidden_context_marker,
|
||||
make_image_placeholder,
|
||||
make_text_block,
|
||||
make_thinking_block,
|
||||
make_tool_result_block,
|
||||
make_tool_use_block,
|
||||
make_unknown_block,
|
||||
render_blocks_to_markdown,
|
||||
)
|
||||
from src.exporters.markdown import MarkdownExporter, _yaml_escape, _format_timestamp
|
||||
from src.exporters.json_export import JSONExporter
|
||||
|
||||
@@ -122,7 +139,7 @@ class TestMarkdownFilenameGeneration:
|
||||
def test_year_in_path(self, tmp_path):
|
||||
exp = MarkdownExporter(tmp_path)
|
||||
path = exp.export(SAMPLE_CONV)
|
||||
assert "/2024/" in str(path)
|
||||
assert ".2024/" in str(path)
|
||||
|
||||
def test_output_structure_provider_project(self, tmp_path):
|
||||
exp = MarkdownExporter(tmp_path, output_structure="provider/project")
|
||||
@@ -250,3 +267,271 @@ class TestFormatTimestamp:
|
||||
|
||||
def test_empty_string(self):
|
||||
assert _format_timestamp("") == ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Block helpers and rendering
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSafeFence:
|
||||
def test_minimum_three_backticks(self):
|
||||
assert _safe_fence("plain text") == "```"
|
||||
|
||||
def test_four_backticks_when_three_in_content(self):
|
||||
assert _safe_fence("here ``` is a fence") == "````"
|
||||
|
||||
def test_five_backticks_when_four_in_content(self):
|
||||
assert _safe_fence("here ```` is four") == "`````"
|
||||
|
||||
def test_handles_empty_string(self):
|
||||
assert _safe_fence("") == "```"
|
||||
|
||||
def test_handles_run_at_end(self):
|
||||
# Trailing run still counted
|
||||
assert _safe_fence("text ending in ```") == "````"
|
||||
|
||||
|
||||
class TestBlockquotePrefix:
|
||||
def test_single_line(self):
|
||||
assert _blockquote_prefix("hello") == "> hello"
|
||||
|
||||
def test_multi_line(self):
|
||||
assert _blockquote_prefix("a\nb\nc") == "> a\n> b\n> c"
|
||||
|
||||
def test_empty_lines_become_naked_quote_marker(self):
|
||||
assert _blockquote_prefix("a\n\nb") == "> a\n>\n> b"
|
||||
|
||||
def test_empty_string(self):
|
||||
assert _blockquote_prefix("") == ">"
|
||||
|
||||
|
||||
class TestBlockConstructors:
|
||||
def test_make_text_block_returns_none_for_empty(self):
|
||||
assert make_text_block("") is None
|
||||
assert make_text_block(" ") is None
|
||||
|
||||
def test_make_text_block_returns_dict(self):
|
||||
b = make_text_block("hello")
|
||||
assert b == {"type": "text", "text": "hello"}
|
||||
|
||||
def test_make_code_block_returns_none_for_empty(self):
|
||||
assert make_code_block("") is None
|
||||
|
||||
def test_make_thinking_block_returns_none_for_empty(self):
|
||||
assert make_thinking_block("") is None
|
||||
|
||||
|
||||
class TestRenderBlocks:
|
||||
def test_text_block_renders_as_paragraph(self):
|
||||
out = render_blocks_to_markdown([make_text_block("Hello world")])
|
||||
assert out == "Hello world"
|
||||
|
||||
def test_blocks_separated_by_blank_line(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[make_text_block("first"), make_text_block("second")]
|
||||
)
|
||||
assert out == "first\n\nsecond"
|
||||
|
||||
def test_code_block_with_language(self):
|
||||
out = render_blocks_to_markdown([make_code_block("print(1)", language="python")])
|
||||
assert "```python" in out
|
||||
assert "print(1)" in out
|
||||
|
||||
def test_thinking_block_uses_blockquote(self):
|
||||
out = render_blocks_to_markdown([make_thinking_block("step 1\nstep 2")])
|
||||
assert "**💭 Reasoning**" in out
|
||||
assert "> step 1" in out
|
||||
assert "> step 2" in out
|
||||
|
||||
def test_tool_use_renders_as_blockquote_with_safe_fence(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[make_tool_use_block("search", {"query": "test"})]
|
||||
)
|
||||
assert "> 🔧 **Tool: search**" in out
|
||||
# Every line of the body is blockquote-prefixed
|
||||
assert "> ```json" in out
|
||||
assert "> }" in out
|
||||
|
||||
def test_tool_use_with_multiline_input(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[make_tool_use_block("complex", {"a": 1, "b": [{"x": "y"}]})]
|
||||
)
|
||||
# Prefix every line of multi-line JSON
|
||||
for line in out.split("\n"):
|
||||
assert line.startswith(">") or line == ""
|
||||
|
||||
def test_tool_result_success_uses_outbox_icon(self):
|
||||
out = render_blocks_to_markdown([make_tool_result_block("OK")])
|
||||
assert "📤 **Result**" in out
|
||||
assert "❌" not in out
|
||||
|
||||
def test_tool_result_error_uses_x_icon(self):
|
||||
out = render_blocks_to_markdown([make_tool_result_block("oops", is_error=True)])
|
||||
assert "❌ **Result (error)**" in out
|
||||
assert "📤" not in out
|
||||
|
||||
def test_tool_result_with_tool_name_in_header(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[make_tool_result_block("done", tool_name="container.exec")]
|
||||
)
|
||||
assert "📤 **Result: container.exec**" in out
|
||||
|
||||
def test_tool_result_error_with_tool_name(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[make_tool_result_block("503", tool_name="web", is_error=True)]
|
||||
)
|
||||
assert "❌ **Result (error): web**" in out
|
||||
|
||||
def test_tool_result_summary_renders_as_italic_line(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[
|
||||
make_tool_result_block(
|
||||
"output",
|
||||
tool_name="container.exec",
|
||||
summary="Reading skill documentation",
|
||||
)
|
||||
]
|
||||
)
|
||||
# Summary line is italic, lives between header and fence,
|
||||
# all inside the blockquote prefix.
|
||||
assert "> *Reading skill documentation*" in out
|
||||
# Order: header before summary before fence
|
||||
header_idx = out.index("Result: container.exec")
|
||||
summary_idx = out.index("Reading skill documentation")
|
||||
fence_idx = out.index("output")
|
||||
assert header_idx < summary_idx < fence_idx
|
||||
|
||||
def test_image_placeholder_rendering(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[make_image_placeholder(ref="file-123", source="user_upload")]
|
||||
)
|
||||
assert "🖼️ **Image attached**" in out
|
||||
assert "`file-123`" in out
|
||||
assert "user_upload" in out
|
||||
assert "content not preserved" in out
|
||||
|
||||
def test_file_placeholder_with_metadata(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[make_file_placeholder(ref="sediment://x", mime="audio/wav", size_bytes=10240, duration_seconds=2.5)]
|
||||
)
|
||||
assert "📎 **File attached**" in out
|
||||
assert "audio/wav" in out
|
||||
assert "KB" in out
|
||||
assert "2.50s" in out
|
||||
|
||||
def test_unknown_block_renders_with_keys(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[
|
||||
make_unknown_block(
|
||||
raw_type="future_x",
|
||||
observed_keys=["foo", "bar"],
|
||||
reason=UNKNOWN_REASON_UNKNOWN_TYPE,
|
||||
)
|
||||
]
|
||||
)
|
||||
assert "⚠️ **Unsupported content**" in out
|
||||
assert "future_x" in out
|
||||
assert "`foo`" in out
|
||||
assert "`bar`" in out
|
||||
|
||||
def test_unknown_extraction_failed_includes_summary(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[
|
||||
make_unknown_block(
|
||||
raw_type="audio_transcription",
|
||||
observed_keys=["asset_pointer"],
|
||||
reason=UNKNOWN_REASON_EXTRACTION_FAILED,
|
||||
summary="expected key 'text' not found",
|
||||
)
|
||||
]
|
||||
)
|
||||
assert "extraction_failed" in out
|
||||
assert "expected key 'text' not found" in out
|
||||
|
||||
def test_hidden_context_marker(self):
|
||||
out = render_blocks_to_markdown(
|
||||
[make_hidden_context_marker("user_editable_context")]
|
||||
)
|
||||
assert "ℹ️ **Hidden context**" in out
|
||||
assert "`user_editable_context`" in out
|
||||
|
||||
def test_safe_fence_prevents_runaway_code_block(self):
|
||||
# Content contains an unbalanced opening fence — without _safe_fence
|
||||
# this would corrupt downstream rendering.
|
||||
evil_content = "before\n```Follow\ntext\nraw is: \"```"
|
||||
block = make_code_block(evil_content)
|
||||
out = render_blocks_to_markdown([block, make_text_block("after")])
|
||||
# The 4-backtick wrap should be present
|
||||
assert "````" in out
|
||||
# The "after" text should appear OUTSIDE any code block — it follows
|
||||
# the closing ```` fence.
|
||||
assert out.endswith("after")
|
||||
|
||||
def test_block_order_preserved(self):
|
||||
blocks = [
|
||||
make_text_block("a"),
|
||||
make_image_placeholder(ref="r1", source="user_upload"),
|
||||
make_text_block("b"),
|
||||
]
|
||||
out = render_blocks_to_markdown(blocks)
|
||||
assert out.index("a") < out.index("Image attached")
|
||||
assert out.index("Image attached") < out.index("b")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Markdown exporter with blocks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
SAMPLE_CONV_BLOCKS = {
|
||||
"id": "blocks12345",
|
||||
"title": "Blocks Conversation",
|
||||
"provider": "claude",
|
||||
"project": None,
|
||||
"created_at": "2024-06-10T14:32:00Z",
|
||||
"updated_at": "2024-06-10T15:00:00Z",
|
||||
"message_count": 1,
|
||||
"messages": [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content_type": "text",
|
||||
"timestamp": None,
|
||||
"blocks": [
|
||||
{"type": "text", "text": "Here is the answer."},
|
||||
{"type": "tool_use", "name": "search", "input": {"q": "x"}, "tool_id": "t1"},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
class TestMarkdownExporterWithBlocks:
|
||||
def test_renders_blocks(self, tmp_path):
|
||||
exp = MarkdownExporter(tmp_path)
|
||||
path = exp.export(SAMPLE_CONV_BLOCKS)
|
||||
body = path.read_text()
|
||||
assert "Here is the answer." in body
|
||||
assert "🔧 **Tool: search**" in body
|
||||
|
||||
def test_falls_back_to_content_when_blocks_missing(self, tmp_path):
|
||||
# Backward-compat: messages with `content` only (no `blocks`) still render.
|
||||
exp = MarkdownExporter(tmp_path)
|
||||
path = exp.export(SAMPLE_CONV) # SAMPLE_CONV has content only, no blocks
|
||||
body = path.read_text()
|
||||
assert "Hello, how are you?" in body
|
||||
|
||||
def test_skips_messages_with_neither_blocks_nor_content(self, tmp_path):
|
||||
conv = {
|
||||
**SAMPLE_CONV_BLOCKS,
|
||||
"messages": [
|
||||
{"role": "user", "content_type": "text", "timestamp": None, "blocks": []},
|
||||
{"role": "assistant", "content_type": "text", "timestamp": None, "blocks": [
|
||||
{"type": "text", "text": "I am here."}
|
||||
]},
|
||||
],
|
||||
}
|
||||
exp = MarkdownExporter(tmp_path)
|
||||
path = exp.export(conv)
|
||||
body = path.read_text()
|
||||
assert "I am here." in body
|
||||
|
||||
+109
-16
@@ -5,7 +5,7 @@ from unittest.mock import MagicMock, patch
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from src.joplin import JoplinClient, JoplinError, _http_error_message, _timeout_message, notebook_title
|
||||
from src.joplin import JoplinClient, JoplinError, _http_error_message, _timeout_message, notebook_path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -31,25 +31,29 @@ def _mock_response(json_data=None, text="", status_code=200):
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# notebook_title helper
|
||||
# notebook_path helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestNotebookTitle:
|
||||
class TestNotebookPath:
|
||||
def test_no_project(self):
|
||||
assert notebook_title("chatgpt", None) == "ChatGPT - No Project"
|
||||
assert notebook_path("chatgpt", None) == ("AI-ChatGPT", "No Project")
|
||||
|
||||
def test_no_project_string(self):
|
||||
assert notebook_title("chatgpt", "no-project") == "ChatGPT - No Project"
|
||||
assert notebook_path("chatgpt", "no-project") == ("AI-ChatGPT", "No Project")
|
||||
|
||||
def test_project_with_hyphens(self):
|
||||
assert notebook_title("chatgpt", "my-project") == "ChatGPT - My Project"
|
||||
assert notebook_path("chatgpt", "my-project") == ("AI-ChatGPT", "My Project")
|
||||
|
||||
def test_claude_provider(self):
|
||||
assert notebook_title("claude", "budget-tracker") == "Claude - Budget Tracker"
|
||||
assert notebook_path("claude", "budget-tracker") == ("AI-Claude", "Budget Tracker")
|
||||
|
||||
def test_multi_word_project(self):
|
||||
assert notebook_title("claude", "ai-research-notes") == "Claude - Ai Research Notes"
|
||||
assert notebook_path("claude", "ai-research-notes") == ("AI-Claude", "Ai Research Notes")
|
||||
|
||||
def test_returns_tuple(self):
|
||||
result = notebook_path("chatgpt", "some-project")
|
||||
assert isinstance(result, tuple) and len(result) == 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -236,18 +240,30 @@ class TestListNotebooks:
|
||||
|
||||
|
||||
class TestGetOrCreateNotebook:
|
||||
def test_returns_existing_notebook_id(self):
|
||||
def test_returns_existing_root_notebook_id(self):
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get:
|
||||
mock_get.return_value = _mock_response(
|
||||
json_data={
|
||||
"items": [{"id": "nb-existing", "title": "ChatGPT - No Project"}],
|
||||
"items": [{"id": "nb-existing", "title": "AI-ChatGPT", "parent_id": ""}],
|
||||
"has_more": False,
|
||||
}
|
||||
)
|
||||
nb_id = client.get_or_create_notebook("ChatGPT - No Project")
|
||||
nb_id = client.get_or_create_notebook("AI-ChatGPT")
|
||||
assert nb_id == "nb-existing"
|
||||
|
||||
def test_returns_existing_child_notebook_id(self):
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get:
|
||||
mock_get.return_value = _mock_response(
|
||||
json_data={
|
||||
"items": [{"id": "nb-child", "title": "No Project", "parent_id": "nb-parent"}],
|
||||
"has_more": False,
|
||||
}
|
||||
)
|
||||
nb_id = client.get_or_create_notebook("No Project", parent_id="nb-parent")
|
||||
assert nb_id == "nb-child"
|
||||
|
||||
def test_creates_new_notebook_when_not_found(self):
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
|
||||
@@ -255,26 +271,103 @@ class TestGetOrCreateNotebook:
|
||||
json_data={"items": [], "has_more": False}
|
||||
)
|
||||
mock_post.return_value = _mock_response(
|
||||
json_data={"id": "nb-new", "title": "ChatGPT - New Project"}
|
||||
json_data={"id": "nb-new", "title": "AI-ChatGPT"}
|
||||
)
|
||||
nb_id = client.get_or_create_notebook("ChatGPT - New Project")
|
||||
nb_id = client.get_or_create_notebook("AI-ChatGPT")
|
||||
assert nb_id == "nb-new"
|
||||
mock_post.assert_called_once()
|
||||
|
||||
def test_creates_child_notebook_with_parent_id(self):
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
|
||||
mock_get.return_value = _mock_response(
|
||||
json_data={"items": [], "has_more": False}
|
||||
)
|
||||
mock_post.return_value = _mock_response(
|
||||
json_data={"id": "nb-child", "title": "My Project"}
|
||||
)
|
||||
nb_id = client.get_or_create_notebook("My Project", parent_id="nb-parent")
|
||||
assert nb_id == "nb-child"
|
||||
_, kwargs = mock_post.call_args
|
||||
assert kwargs["json"]["parent_id"] == "nb-parent"
|
||||
|
||||
def test_does_not_include_parent_id_for_root(self):
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
|
||||
mock_get.return_value = _mock_response(json_data={"items": [], "has_more": False})
|
||||
mock_post.return_value = _mock_response(json_data={"id": "nb-root", "title": "AI-Claude"})
|
||||
client.get_or_create_notebook("AI-Claude")
|
||||
_, kwargs = mock_post.call_args
|
||||
assert "parent_id" not in kwargs["json"]
|
||||
|
||||
def test_caches_notebook_after_first_load(self):
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get:
|
||||
mock_get.return_value = _mock_response(
|
||||
json_data={
|
||||
"items": [{"id": "nb1", "title": "Claude - No Project"}],
|
||||
"items": [{"id": "nb1", "title": "AI-Claude", "parent_id": ""}],
|
||||
"has_more": False,
|
||||
}
|
||||
)
|
||||
# Call twice — GET /folders should only happen once
|
||||
client.get_or_create_notebook("Claude - No Project")
|
||||
client.get_or_create_notebook("Claude - No Project")
|
||||
client.get_or_create_notebook("AI-Claude")
|
||||
client.get_or_create_notebook("AI-Claude")
|
||||
assert mock_get.call_count == 1
|
||||
|
||||
def test_different_parent_ids_are_distinct_cache_entries(self):
|
||||
"""Same title under different parents are different notebooks."""
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get:
|
||||
mock_get.return_value = _mock_response(
|
||||
json_data={
|
||||
"items": [
|
||||
{"id": "nb-a", "title": "No Project", "parent_id": "parent-chatgpt"},
|
||||
{"id": "nb-b", "title": "No Project", "parent_id": "parent-claude"},
|
||||
],
|
||||
"has_more": False,
|
||||
}
|
||||
)
|
||||
id_a = client.get_or_create_notebook("No Project", parent_id="parent-chatgpt")
|
||||
id_b = client.get_or_create_notebook("No Project", parent_id="parent-claude")
|
||||
assert id_a == "nb-a"
|
||||
assert id_b == "nb-b"
|
||||
|
||||
|
||||
class TestGetOrCreateNotebookPath:
|
||||
def test_creates_two_level_path(self):
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
|
||||
mock_get.return_value = _mock_response(json_data={"items": [], "has_more": False})
|
||||
mock_post.side_effect = [
|
||||
_mock_response(json_data={"id": "nb-parent", "title": "AI-ChatGPT"}),
|
||||
_mock_response(json_data={"id": "nb-child", "title": "No Project"}),
|
||||
]
|
||||
leaf_id = client.get_or_create_notebook_path(["AI-ChatGPT", "No Project"])
|
||||
assert leaf_id == "nb-child"
|
||||
assert mock_post.call_count == 2
|
||||
# Second POST should use the parent's ID
|
||||
_, kwargs = mock_post.call_args_list[1]
|
||||
assert kwargs["json"]["parent_id"] == "nb-parent"
|
||||
|
||||
def test_reuses_existing_parent_for_new_child(self):
|
||||
client = _make_client()
|
||||
with patch("requests.get") as mock_get, patch("requests.post") as mock_post:
|
||||
mock_get.return_value = _mock_response(
|
||||
json_data={
|
||||
"items": [{"id": "nb-parent", "title": "AI-Claude", "parent_id": ""}],
|
||||
"has_more": False,
|
||||
}
|
||||
)
|
||||
mock_post.return_value = _mock_response(
|
||||
json_data={"id": "nb-child", "title": "Budget Tracker"}
|
||||
)
|
||||
leaf_id = client.get_or_create_notebook_path(["AI-Claude", "Budget Tracker"])
|
||||
assert leaf_id == "nb-child"
|
||||
# Only one POST — the parent already existed
|
||||
assert mock_post.call_count == 1
|
||||
_, kwargs = mock_post.call_args
|
||||
assert kwargs["json"]["parent_id"] == "nb-parent"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# create_note
|
||||
|
||||
@@ -0,0 +1,248 @@
|
||||
"""Tests for media downloads and Joplin resource rewriting."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from src.blocks import (
|
||||
make_file_placeholder,
|
||||
make_image_placeholder,
|
||||
render_blocks_to_markdown,
|
||||
)
|
||||
from src.loss_report import LossReport
|
||||
from src.media import resolve_media, resolve_media_policy
|
||||
from src.providers.base import ProviderError
|
||||
from src.providers.chatgpt import parse_asset_file_id
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Asset reference parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestParseAssetFileId:
|
||||
def test_plain_sediment(self):
|
||||
assert parse_asset_file_id("sediment://file_00000000245c71fda5") == "file_00000000245c71fda5"
|
||||
|
||||
def test_generated_image_with_hash_and_page(self):
|
||||
ref = "sediment://8456107fc383a53#file_00000000979c71f685#p_6.png"
|
||||
assert parse_asset_file_id(ref) == "file_00000000979c71f685"
|
||||
|
||||
def test_file_service_scheme(self):
|
||||
assert parse_asset_file_id("file-service://file-AbCdEf") == "file-AbCdEf"
|
||||
|
||||
def test_unrecognised(self):
|
||||
assert parse_asset_file_id("https://example.com/x.png") is None
|
||||
assert parse_asset_file_id("") is None
|
||||
assert parse_asset_file_id(None) is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Media policy
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestMediaPolicy:
|
||||
def test_default(self, monkeypatch):
|
||||
monkeypatch.delenv("EXPORTER_DOWNLOAD_MEDIA", raising=False)
|
||||
assert resolve_media_policy() == "images"
|
||||
|
||||
def test_valid(self, monkeypatch):
|
||||
monkeypatch.setenv("EXPORTER_DOWNLOAD_MEDIA", "all")
|
||||
assert resolve_media_policy() == "all"
|
||||
|
||||
def test_invalid_falls_back(self, monkeypatch, caplog):
|
||||
monkeypatch.setenv("EXPORTER_DOWNLOAD_MEDIA", "bogus")
|
||||
assert resolve_media_policy() == "images"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# resolve_media
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeProvider:
|
||||
"""Minimal provider exposing download_asset + the ref parser."""
|
||||
|
||||
def __init__(self, assets=None, fail_refs=None):
|
||||
self._assets = assets or {}
|
||||
self._fail_refs = fail_refs or {}
|
||||
self.calls = []
|
||||
|
||||
def parse_asset_file_id(self, ref):
|
||||
return parse_asset_file_id(ref)
|
||||
|
||||
def download_asset(self, ref):
|
||||
self.calls.append(ref)
|
||||
if ref in self._fail_refs:
|
||||
raise ProviderError("chatgpt", "download_asset", self._fail_refs[ref])
|
||||
return self._assets[ref] # (content, mime, file_name)
|
||||
|
||||
|
||||
def _conv_with(blocks):
|
||||
return {
|
||||
"id": "conv-1",
|
||||
"title": "Has Media",
|
||||
"provider": "chatgpt",
|
||||
"project": None,
|
||||
"created_at": "2026-05-20T00:00:00+00:00",
|
||||
"messages": [{"role": "user", "blocks": blocks}],
|
||||
}
|
||||
|
||||
|
||||
class TestResolveMedia:
|
||||
def test_downloads_image_and_inlines(self, tmp_path):
|
||||
ref = "sediment://file_img1"
|
||||
provider = _FakeProvider({ref: (b"\x89PNG\r\n", "image/png", "x.png")})
|
||||
block = make_image_placeholder(ref=ref, source="user_upload")
|
||||
conv = _conv_with([block])
|
||||
report = LossReport()
|
||||
|
||||
n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
|
||||
|
||||
assert n == 1
|
||||
assert report.media_downloaded == 1
|
||||
assert block["local_path"] == "media/file_img1.png"
|
||||
rendered = render_blocks_to_markdown([block])
|
||||
assert rendered == ""
|
||||
# File written under the conversation's media/ dir
|
||||
written = list(tmp_path.rglob("media/file_img1.png"))
|
||||
assert written and written[0].read_bytes() == b"\x89PNG\r\n"
|
||||
|
||||
def test_images_policy_skips_files(self, tmp_path):
|
||||
ref = "sediment://file_audio1"
|
||||
provider = _FakeProvider({ref: (b"RIFF", "audio/wav", "a.wav")})
|
||||
block = make_file_placeholder(ref=ref, mime="audio/wav", size_bytes=1000)
|
||||
conv = _conv_with([block])
|
||||
report = LossReport()
|
||||
|
||||
n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
|
||||
assert n == 0
|
||||
assert "local_path" not in block
|
||||
assert provider.calls == []
|
||||
|
||||
def test_all_policy_downloads_files(self, tmp_path):
|
||||
ref = "sediment://file_audio1"
|
||||
provider = _FakeProvider({ref: (b"RIFFdata", "audio/wav", "a.wav")})
|
||||
block = make_file_placeholder(ref=ref, mime="audio/wav", size_bytes=8)
|
||||
conv = _conv_with([block])
|
||||
report = LossReport()
|
||||
|
||||
n = resolve_media(conv, provider, tmp_path, "provider/project/year", "all", report)
|
||||
assert n == 1
|
||||
assert block["local_path"] == "media/file_audio1.wav"
|
||||
rendered = render_blocks_to_markdown([block])
|
||||
assert "media/file_audio1.wav" in rendered and rendered.startswith("> 📎")
|
||||
|
||||
def test_off_policy_noop(self, tmp_path):
|
||||
provider = _FakeProvider({"sediment://file_x": (b"x", "image/png", None)})
|
||||
block = make_image_placeholder(ref="sediment://file_x", source="user_upload")
|
||||
conv = _conv_with([block])
|
||||
report = LossReport()
|
||||
assert resolve_media(conv, provider, tmp_path, "provider/project/year", "off", report) == 0
|
||||
assert provider.calls == []
|
||||
|
||||
def test_idempotent_uses_disk(self, tmp_path):
|
||||
ref = "sediment://file_img1"
|
||||
provider = _FakeProvider({ref: (b"\x89PNG", "image/png", "x.png")})
|
||||
conv = _conv_with([make_image_placeholder(ref=ref, source="user_upload")])
|
||||
report = LossReport()
|
||||
resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
|
||||
assert len(provider.calls) == 1
|
||||
|
||||
# Second run, fresh blocks: file already on disk → no new API call.
|
||||
conv2 = _conv_with([make_image_placeholder(ref=ref, source="user_upload")])
|
||||
resolve_media(conv2, provider, tmp_path, "provider/project/year", "images", LossReport())
|
||||
assert len(provider.calls) == 1 # unchanged
|
||||
assert conv2["messages"][0]["blocks"][0]["local_path"] == "media/file_img1.png"
|
||||
|
||||
def test_failure_keeps_placeholder_and_counts(self, tmp_path):
|
||||
ref = "sediment://file_gone"
|
||||
provider = _FakeProvider(fail_refs={ref: RuntimeError("Signed URL returned HTTP 404")})
|
||||
block = make_image_placeholder(ref=ref, source="model_generated")
|
||||
conv = _conv_with([block])
|
||||
report = LossReport()
|
||||
|
||||
n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
|
||||
assert n == 0
|
||||
assert "local_path" not in block
|
||||
assert report.media_failed["expired-or-missing"] == 1
|
||||
# Still renders as a placeholder, not a broken image link
|
||||
assert render_blocks_to_markdown([block]).startswith("> 🖼️")
|
||||
|
||||
def test_provider_without_download_asset(self, tmp_path):
|
||||
"""claude-code has no remote assets — resolve_media must no-op."""
|
||||
class NoDownload:
|
||||
pass
|
||||
block = make_image_placeholder(ref="sediment://file_x", source="user_upload")
|
||||
conv = _conv_with([block])
|
||||
assert resolve_media(conv, NoDownload(), tmp_path, "provider/project/year", "images", LossReport()) == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Joplin media rewriting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _FakeJoplin:
|
||||
def __init__(self):
|
||||
self.uploaded = []
|
||||
self._n = 0
|
||||
|
||||
def create_resource(self, file_path, title=None):
|
||||
self.uploaded.append(Path(file_path).name)
|
||||
self._n += 1
|
||||
return f"res{self._n}"
|
||||
|
||||
|
||||
class TestUploadMediaAndRewrite:
|
||||
def _note(self, tmp_path, body):
|
||||
media = tmp_path / "media"
|
||||
media.mkdir()
|
||||
(media / "file_img1.png").write_bytes(b"\x89PNG")
|
||||
(media / "clip.wav").write_bytes(b"RIFF")
|
||||
return body
|
||||
|
||||
def test_rewrites_image_and_file_links(self, tmp_path):
|
||||
from src.joplin import upload_media_and_rewrite
|
||||
body = self._note(
|
||||
tmp_path,
|
||||
"Look: \n"
|
||||
"> 📎 **File attached** — [clip.wav](media/clip.wav) (audio/wav)",
|
||||
)
|
||||
client = _FakeJoplin()
|
||||
new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, {})
|
||||
|
||||
assert "" in new_body
|
||||
assert "(:/res2)" in new_body
|
||||
assert "media/" not in new_body
|
||||
assert res_map == {"media/file_img1.png": "res1", "media/clip.wav": "res2"}
|
||||
assert len(client.uploaded) == 2
|
||||
|
||||
def test_reuses_known_resource_ids(self, tmp_path):
|
||||
from src.joplin import upload_media_and_rewrite
|
||||
body = self._note(tmp_path, "")
|
||||
client = _FakeJoplin()
|
||||
existing = {"media/file_img1.png": "existing-res"}
|
||||
new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, existing)
|
||||
assert "(:/existing-res)" in new_body
|
||||
assert client.uploaded == [] # no re-upload
|
||||
assert res_map == existing
|
||||
|
||||
def test_missing_file_left_as_link(self, tmp_path):
|
||||
from src.joplin import upload_media_and_rewrite
|
||||
(tmp_path / "media").mkdir()
|
||||
body = ""
|
||||
client = _FakeJoplin()
|
||||
new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, {})
|
||||
assert new_body == body
|
||||
assert res_map == {}
|
||||
assert client.uploaded == []
|
||||
|
||||
def test_no_media_links_untouched(self, tmp_path):
|
||||
from src.joplin import upload_media_and_rewrite
|
||||
body = "Just text with [a link](https://example.com) and `media/foo` in code."
|
||||
client = _FakeJoplin()
|
||||
new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, {})
|
||||
assert new_body == body
|
||||
assert res_map == {}
|
||||
+731
-65
@@ -1,19 +1,53 @@
|
||||
"""Unit tests for src/providers/ using fixture files."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from src.blocks import (
|
||||
BLOCK_TYPE_FILE_PLACEHOLDER,
|
||||
BLOCK_TYPE_HIDDEN_CONTEXT_MARKER,
|
||||
BLOCK_TYPE_IMAGE_PLACEHOLDER,
|
||||
BLOCK_TYPE_TEXT,
|
||||
BLOCK_TYPE_THINKING,
|
||||
BLOCK_TYPE_TOOL_RESULT,
|
||||
BLOCK_TYPE_TOOL_USE,
|
||||
BLOCK_TYPE_UNKNOWN,
|
||||
render_blocks_to_markdown,
|
||||
)
|
||||
from src.loss_report import LossReport
|
||||
|
||||
FIXTURES = Path(__file__).parent / "fixtures"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _block_types(message: dict) -> list[str]:
|
||||
return [b.get("type") for b in (message.get("blocks") or [])]
|
||||
|
||||
|
||||
def _first_block(message: dict, block_type: str) -> dict | None:
|
||||
for b in message.get("blocks") or []:
|
||||
if b.get("type") == block_type:
|
||||
return b
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ChatGPT
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestChatGPTNormalization:
|
||||
"""Test ChatGPTProvider.normalize_conversation() using fixture data."""
|
||||
"""ChatGPT normalize_conversation block-extraction behavior."""
|
||||
|
||||
def _get_provider(self):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
# Bypass __init__ token check
|
||||
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||
import requests
|
||||
p._session = requests.Session()
|
||||
@@ -31,7 +65,6 @@ class TestChatGPTNormalization:
|
||||
assert result["id"] == "chatgpt-conv-001"
|
||||
assert result["title"] == "Python Async Tutorial"
|
||||
assert result["provider"] == "chatgpt"
|
||||
# No entry in _project_map → project is None
|
||||
assert result["project"] is None
|
||||
assert result["created_at"] != ""
|
||||
assert result["updated_at"] != ""
|
||||
@@ -46,7 +79,6 @@ class TestChatGPTNormalization:
|
||||
assert result["id"] == "chatgpt-conv-002"
|
||||
|
||||
def test_normalizes_with_project_from_map(self):
|
||||
"""Project name from _project_map (populated by fetch_all_conversations) flows through."""
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
p._project_map["chatgpt-conv-001"] = "My Research Project"
|
||||
@@ -54,32 +86,169 @@ class TestChatGPTNormalization:
|
||||
|
||||
assert result["project"] == "My Research Project"
|
||||
|
||||
def test_extracts_text_messages(self):
|
||||
def test_text_message_emits_text_block(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
assert len(result["messages"]) >= 2
|
||||
user_msgs = [m for m in result["messages"] if m["role"] == "user"]
|
||||
assert any("async" in m["content"].lower() for m in user_msgs)
|
||||
# The "How does async/await..." message
|
||||
async_msgs = [
|
||||
m for m in user_msgs
|
||||
if any(
|
||||
"async" in (b.get("text") or "").lower()
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert async_msgs, "expected a user message about async/await"
|
||||
assert _block_types(async_msgs[0]) == [BLOCK_TYPE_TEXT]
|
||||
|
||||
def test_skips_non_text_content_with_warning(self, caplog):
|
||||
import logging
|
||||
def test_code_block_preserved_with_language(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
with caplog.at_level(logging.WARNING):
|
||||
result = p.normalize_conversation(raw)
|
||||
# The fixture has an image_asset_pointer node — should be warned about
|
||||
assert any(
|
||||
"image_asset_pointer" in r.message or "rich content" in r.message
|
||||
for r in caplog.records
|
||||
)
|
||||
|
||||
def test_model_editable_context_included_without_warning(self, caplog):
|
||||
"""model_editable_context messages (project instructions) should be included, not warned about."""
|
||||
import logging
|
||||
conv = {
|
||||
"id": "test-conv-mec",
|
||||
assistant_msgs = [m for m in result["messages"] if m["role"] == "assistant"]
|
||||
# The first assistant message is the async/await answer with a python fence
|
||||
text_block = _first_block(assistant_msgs[0], BLOCK_TYPE_TEXT)
|
||||
assert text_block is not None
|
||||
assert "```python" in text_block["text"]
|
||||
|
||||
def test_multimodal_voice_user_message(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
# node-mm-user: audio_transcription "What is the capital of France?"
|
||||
# + real_time_user_audio_video_asset_pointer wrapping a sediment:// URL
|
||||
capital_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
"capital of france" in (b.get("text") or "").lower()
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert capital_msgs, "expected the audio_transcription text to surface"
|
||||
types = _block_types(capital_msgs[0])
|
||||
assert BLOCK_TYPE_TEXT in types
|
||||
assert BLOCK_TYPE_FILE_PLACEHOLDER in types
|
||||
|
||||
file_block = _first_block(capital_msgs[0], BLOCK_TYPE_FILE_PLACEHOLDER)
|
||||
assert file_block["ref"].startswith("sediment://")
|
||||
assert file_block["mime"] == "audio/wav"
|
||||
assert file_block["size_bytes"] == 50000
|
||||
assert file_block["duration_seconds"] == pytest.approx(2.5)
|
||||
|
||||
def test_multimodal_voice_reverse_order_preserved(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
# node-mm-user-rev has parts in REVERSE order: asset first, transcription second.
|
||||
rev_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
"tell me more" in (b.get("text") or "").lower()
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert rev_msgs, "expected the reverse-order voice message"
|
||||
types = _block_types(rev_msgs[0])
|
||||
# Order preserved: file_placeholder before text
|
||||
assert types == [BLOCK_TYPE_FILE_PLACEHOLDER, BLOCK_TYPE_TEXT]
|
||||
|
||||
def test_image_only_user_message_renders(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
image_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(b.get("type") == BLOCK_TYPE_IMAGE_PLACEHOLDER for b in (m.get("blocks") or []))
|
||||
]
|
||||
assert image_msgs, "image-only user message should now render"
|
||||
|
||||
def test_user_editable_context_emits_blocks(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
p._hidden_content = "full"
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
# The user_editable_context message has user_profile + user_instructions.
|
||||
# It should now appear (was silently dropped pre-v0.4.0).
|
||||
uec_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
"Custom Instructions" in (b.get("text") or "")
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert uec_msgs, "user_editable_context should be visible in output"
|
||||
# Hidden context marker should be prepended.
|
||||
assert uec_msgs[0]["blocks"][0]["type"] == BLOCK_TYPE_HIDDEN_CONTEXT_MARKER
|
||||
|
||||
def test_user_editable_context_uses_safe_fence(self):
|
||||
"""The user_instructions value contains embedded triple-backticks; the rendered
|
||||
Markdown must use a fence longer than 3 backticks so embedded fences are inert.
|
||||
"""
|
||||
from src.blocks import render_blocks_to_markdown
|
||||
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
p._hidden_content = "full"
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
uec_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
"Custom Instructions" in (b.get("text") or "")
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert uec_msgs
|
||||
rendered = render_blocks_to_markdown(uec_msgs[0]["blocks"])
|
||||
# Content has ``` inside, so the wrap fence must be at least 4 backticks.
|
||||
assert "````" in rendered, "expected a 4+ backtick safe-fence wrap"
|
||||
|
||||
def test_message_roles_are_valid(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
for msg in result["messages"]:
|
||||
assert msg["role"] in ("user", "assistant", "system", "tool")
|
||||
|
||||
def test_message_count_matches(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
assert result["message_count"] == len(result["messages"])
|
||||
|
||||
def test_loss_report_records_messages(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
report = LossReport()
|
||||
result = p.normalize_conversation(raw, report)
|
||||
assert report.messages_rendered == len(result["messages"])
|
||||
assert report.conversations == 1
|
||||
|
||||
|
||||
class TestChatGPTUnknownContent:
|
||||
"""Unrecognised content types should produce visible unknown blocks + WARNING + tally."""
|
||||
|
||||
def _get_provider(self):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||
import requests
|
||||
p._session = requests.Session()
|
||||
p._org_id = None
|
||||
p._project_ids = []
|
||||
p._project_map = {}
|
||||
p._project_name_cache = {}
|
||||
return p
|
||||
|
||||
def _make_unknown_conv(self):
|
||||
return {
|
||||
"id": "test-unknown",
|
||||
"title": "Test",
|
||||
"create_time": 1700000000.0,
|
||||
"update_time": 1700000001.0,
|
||||
@@ -91,46 +260,275 @@ class TestChatGPTNormalization:
|
||||
"id": "msg1",
|
||||
"author": {"role": "user"},
|
||||
"content": {
|
||||
"content_type": "model_editable_context",
|
||||
"parts": ["These are the project instructions."],
|
||||
"content_type": "future_unknown_type_xyz",
|
||||
"some_field": "value",
|
||||
},
|
||||
"create_time": 1700000001.0,
|
||||
"status": "finished_successfully",
|
||||
},
|
||||
"parent": "root",
|
||||
"children": [],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
def test_unknown_content_type_produces_unknown_block(self):
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(self._make_unknown_conv())
|
||||
assert any(
|
||||
b.get("type") == BLOCK_TYPE_UNKNOWN
|
||||
for m in result["messages"]
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
|
||||
def test_unknown_content_type_logs_warning(self, caplog):
|
||||
p = self._get_provider()
|
||||
with caplog.at_level(logging.WARNING):
|
||||
result = p.normalize_conversation(conv)
|
||||
assert any(m["content"] == "These are the project instructions." for m in result["messages"])
|
||||
assert not any("model_editable_context" in r.message for r in caplog.records)
|
||||
p.normalize_conversation(self._make_unknown_conv())
|
||||
assert any("future_unknown_type_xyz" in r.message for r in caplog.records)
|
||||
|
||||
def test_message_roles_are_valid(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
def test_unknown_content_type_increments_loss_report(self):
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
for msg in result["messages"]:
|
||||
assert msg["role"] in ("user", "assistant", "system")
|
||||
report = LossReport()
|
||||
p.normalize_conversation(self._make_unknown_conv(), report)
|
||||
assert report.unknown_blocks["future_unknown_type_xyz"] == 1
|
||||
|
||||
def test_message_count_matches(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
assert result["message_count"] == len(result["messages"])
|
||||
|
||||
def test_code_fence_preserved(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
all_content = " ".join(m["content"] for m in result["messages"])
|
||||
assert "```python" in all_content
|
||||
class TestChatGPTHiddenContentPolicy:
|
||||
"""EXPORTER_HIDDEN_CONTENT: collapse retrieval dumps and hidden context.
|
||||
|
||||
file_search dumps re-inject attached-file text on every tool run (measured
|
||||
86% of content bytes on real data) and are NOT hidden-flagged — author.name
|
||||
is the discriminator. Hidden-flagged messages (Custom Instructions) are the
|
||||
secondary case.
|
||||
"""
|
||||
|
||||
def _get_provider(self, policy=None):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||
import requests
|
||||
p._session = requests.Session()
|
||||
p._org_id = None
|
||||
p._project_ids = []
|
||||
p._project_map = {}
|
||||
p._project_name_cache = {}
|
||||
if policy is not None:
|
||||
p._hidden_content = policy
|
||||
return p
|
||||
|
||||
def _make_conv(self):
|
||||
return {
|
||||
"id": "test-collapse",
|
||||
"title": "T",
|
||||
"create_time": 1700000000.0,
|
||||
"update_time": 1700000001.0,
|
||||
"mapping": {
|
||||
"root": {"id": "root", "message": None, "parent": None, "children": ["dump"]},
|
||||
"dump": {
|
||||
"id": "dump",
|
||||
"parent": "root",
|
||||
"children": ["answer"],
|
||||
"message": {
|
||||
"id": "dump",
|
||||
"author": {"role": "tool", "name": "file_search"},
|
||||
"content": {
|
||||
"content_type": "multimodal_text",
|
||||
"parts": ["Full text of an attached file " * 100],
|
||||
},
|
||||
},
|
||||
},
|
||||
"answer": {
|
||||
"id": "answer",
|
||||
"parent": "dump",
|
||||
"children": ["uec"],
|
||||
"message": {
|
||||
"id": "answer",
|
||||
"author": {"role": "assistant"},
|
||||
"content": {"content_type": "text", "parts": ["The answer."]},
|
||||
},
|
||||
},
|
||||
"uec": {
|
||||
"id": "uec",
|
||||
"parent": "answer",
|
||||
"children": [],
|
||||
"message": {
|
||||
"id": "uec",
|
||||
"author": {"role": "user"},
|
||||
"content": {
|
||||
"content_type": "user_editable_context",
|
||||
"user_profile": "Preferred name: Jesse",
|
||||
"user_instructions": "Be concise.",
|
||||
},
|
||||
"metadata": {"is_visually_hidden_from_conversation": True},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
def test_default_policy_is_placeholder(self, monkeypatch):
|
||||
from src.blocks import BLOCK_TYPE_COLLAPSED
|
||||
monkeypatch.delenv("EXPORTER_HIDDEN_CONTENT", raising=False)
|
||||
# No _hidden_content attribute → env fallback → placeholder default.
|
||||
result = self._get_provider().normalize_conversation(self._make_conv())
|
||||
collapsed = [
|
||||
b for m in result["messages"] for b in m["blocks"]
|
||||
if b.get("type") == BLOCK_TYPE_COLLAPSED
|
||||
]
|
||||
assert len(collapsed) == 2
|
||||
|
||||
def test_placeholder_collapses_file_search_dump(self):
|
||||
from src.blocks import BLOCK_TYPE_COLLAPSED, COLLAPSED_KIND_TOOL_DUMP
|
||||
result = self._get_provider("placeholder").normalize_conversation(self._make_conv())
|
||||
tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
|
||||
assert len(tool_msgs) == 1
|
||||
(block,) = tool_msgs[0]["blocks"]
|
||||
assert block["type"] == BLOCK_TYPE_COLLAPSED
|
||||
assert block["kind"] == COLLAPSED_KIND_TOOL_DUMP
|
||||
assert block["origin"] == "file_search"
|
||||
assert block["size_bytes"] > 1000
|
||||
|
||||
def test_placeholder_collapses_hidden_flagged_message(self):
|
||||
from src.blocks import BLOCK_TYPE_COLLAPSED, COLLAPSED_KIND_HIDDEN_CONTEXT
|
||||
result = self._get_provider("placeholder").normalize_conversation(self._make_conv())
|
||||
user_msgs = [m for m in result["messages"] if m["role"] == "user"]
|
||||
assert len(user_msgs) == 1
|
||||
(block,) = user_msgs[0]["blocks"]
|
||||
assert block["type"] == BLOCK_TYPE_COLLAPSED
|
||||
assert block["kind"] == COLLAPSED_KIND_HIDDEN_CONTEXT
|
||||
assert block["origin"] == "user_editable_context"
|
||||
|
||||
def test_placeholder_keeps_normal_messages(self):
|
||||
result = self._get_provider("placeholder").normalize_conversation(self._make_conv())
|
||||
assistant_msgs = [m for m in result["messages"] if m["role"] == "assistant"]
|
||||
assert assistant_msgs[0]["blocks"][0]["type"] == BLOCK_TYPE_TEXT
|
||||
assert assistant_msgs[0]["blocks"][0]["text"] == "The answer."
|
||||
|
||||
def test_non_retrieval_tool_authors_not_collapsed(self):
|
||||
"""web.run / browser / python tool messages are real content — keep."""
|
||||
from src.blocks import BLOCK_TYPE_COLLAPSED
|
||||
conv = self._make_conv()
|
||||
conv["mapping"]["dump"]["message"]["author"]["name"] = "web.run"
|
||||
conv["mapping"]["dump"]["message"]["content"] = {
|
||||
"content_type": "text",
|
||||
"parts": ["Search results summary."],
|
||||
}
|
||||
result = self._get_provider("placeholder").normalize_conversation(conv)
|
||||
tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
|
||||
assert tool_msgs[0]["blocks"][0]["type"] != BLOCK_TYPE_COLLAPSED
|
||||
|
||||
def test_omit_drops_messages_entirely(self):
|
||||
result = self._get_provider("omit").normalize_conversation(self._make_conv())
|
||||
assert [m["role"] for m in result["messages"]] == ["assistant"]
|
||||
|
||||
def test_full_preserves_v040_behavior(self):
|
||||
result = self._get_provider("full").normalize_conversation(self._make_conv())
|
||||
tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
|
||||
assert tool_msgs[0]["blocks"][0]["type"] == BLOCK_TYPE_TEXT
|
||||
user_msgs = [m for m in result["messages"] if m["role"] == "user"]
|
||||
assert user_msgs[0]["blocks"][0]["type"] == BLOCK_TYPE_HIDDEN_CONTEXT_MARKER
|
||||
|
||||
def test_collapsed_tallied_in_loss_report(self):
|
||||
report = LossReport()
|
||||
self._get_provider("placeholder").normalize_conversation(self._make_conv(), report)
|
||||
assert report.collapsed["file_search"] == 1
|
||||
assert report.collapsed["user_editable_context"] == 1
|
||||
assert report.collapsed_bytes > 1000
|
||||
summary = report.format_summary()
|
||||
assert "collapsed by policy: 2" in summary
|
||||
assert "EXPORTER_HIDDEN_CONTENT=full to keep" in summary
|
||||
|
||||
def test_collapsed_block_renders_one_line_with_size(self):
|
||||
from src.blocks import BLOCK_TYPE_COLLAPSED
|
||||
result = self._get_provider("placeholder").normalize_conversation(self._make_conv())
|
||||
tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
|
||||
rendered = render_blocks_to_markdown(tool_msgs[0]["blocks"])
|
||||
assert rendered.startswith("> 🔧 **Tool output** — `file_search` (")
|
||||
assert "KB" in rendered
|
||||
assert "omitted (EXPORTER_HIDDEN_CONTENT=full to keep)" in rendered
|
||||
assert "\n" not in rendered
|
||||
|
||||
def test_invalid_env_value_falls_back_to_placeholder(self, monkeypatch, caplog):
|
||||
from src.providers.chatgpt import resolve_hidden_content_policy
|
||||
monkeypatch.setenv("EXPORTER_HIDDEN_CONTENT", "bogus")
|
||||
with caplog.at_level(logging.WARNING):
|
||||
assert resolve_hidden_content_policy() == "placeholder"
|
||||
assert any("EXPORTER_HIDDEN_CONTENT" in r.message for r in caplog.records)
|
||||
|
||||
|
||||
class TestRequestPacing:
|
||||
"""REQUEST_DELAY politeness pacing between consecutive API requests."""
|
||||
|
||||
def _get_provider(self, delay):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||
p._request_delay = delay
|
||||
p._last_request_at = None
|
||||
return p
|
||||
|
||||
def test_first_request_not_delayed(self, monkeypatch):
|
||||
sleeps = []
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: sleeps.append(s))
|
||||
p = self._get_provider(2.0)
|
||||
p._pace()
|
||||
assert sleeps == []
|
||||
|
||||
def test_consecutive_requests_paced_with_jitter_window(self, monkeypatch):
|
||||
sleeps = []
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: sleeps.append(s))
|
||||
monkeypatch.setattr("src.providers.base.random.uniform", lambda a, b: 1.0)
|
||||
clock = {"now": 100.0}
|
||||
monkeypatch.setattr("src.providers.base.time.monotonic", lambda: clock["now"])
|
||||
p = self._get_provider(2.0)
|
||||
p._pace()
|
||||
clock["now"] = 100.5 # only 0.5s elapsed since last request
|
||||
p._pace()
|
||||
assert sleeps == [pytest.approx(1.5)]
|
||||
|
||||
def test_no_sleep_when_enough_time_elapsed(self, monkeypatch):
|
||||
sleeps = []
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: sleeps.append(s))
|
||||
monkeypatch.setattr("src.providers.base.random.uniform", lambda a, b: 1.0)
|
||||
clock = {"now": 100.0}
|
||||
monkeypatch.setattr("src.providers.base.time.monotonic", lambda: clock["now"])
|
||||
p = self._get_provider(1.0)
|
||||
p._pace()
|
||||
clock["now"] = 105.0
|
||||
p._pace()
|
||||
assert sleeps == []
|
||||
|
||||
def test_zero_delay_disables_pacing(self, monkeypatch):
|
||||
sleeps = []
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: sleeps.append(s))
|
||||
p = self._get_provider(0)
|
||||
p._pace()
|
||||
p._pace()
|
||||
assert sleeps == []
|
||||
|
||||
def test_pace_noop_without_init_attrs(self):
|
||||
"""Providers built via __new__ in tests have no pacing attrs — must not crash."""
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||
p._pace() # no exception
|
||||
|
||||
def test_resolve_request_delay_default_and_overrides(self, monkeypatch, caplog):
|
||||
from src.providers.base import resolve_request_delay, DEFAULT_REQUEST_DELAY
|
||||
monkeypatch.delenv("REQUEST_DELAY", raising=False)
|
||||
assert resolve_request_delay() == DEFAULT_REQUEST_DELAY
|
||||
monkeypatch.setenv("REQUEST_DELAY", "2.5")
|
||||
assert resolve_request_delay() == 2.5
|
||||
monkeypatch.setenv("REQUEST_DELAY", "0")
|
||||
assert resolve_request_delay() == 0.0
|
||||
monkeypatch.setenv("REQUEST_DELAY", "abc")
|
||||
with caplog.at_level(logging.WARNING):
|
||||
assert resolve_request_delay() == DEFAULT_REQUEST_DELAY
|
||||
assert any("REQUEST_DELAY" in r.message for r in caplog.records)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Claude
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestClaudeNormalization:
|
||||
"""Test ClaudeProvider.normalize_conversation() using fixture data."""
|
||||
"""Claude normalize_conversation block-extraction behavior."""
|
||||
|
||||
def _get_provider(self):
|
||||
from src.providers.claude import ClaudeProvider
|
||||
@@ -150,55 +548,138 @@ class TestClaudeNormalization:
|
||||
assert result["provider"] == "claude"
|
||||
assert result["project"] == "StarTOS Packaging"
|
||||
assert result["created_at"] == "2024-06-10T14:32:00.000Z"
|
||||
assert isinstance(result["messages"], list)
|
||||
|
||||
def test_normalizes_without_project(self):
|
||||
raw = json.loads((FIXTURES / "claude_no_project.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
assert result["project"] is None
|
||||
assert result["id"] == "claude-conv-002"
|
||||
|
||||
def test_string_content_extracted(self):
|
||||
raw = json.loads((FIXTURES / "claude_no_project.json").read_text())
|
||||
def test_string_content_emits_text_block(self):
|
||||
raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
assert any("Docker" in m["content"] for m in result["messages"])
|
||||
thanks_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
"thank you" in (b.get("text") or "").lower()
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert thanks_msgs
|
||||
|
||||
def test_list_content_extracted(self):
|
||||
def test_list_content_emits_blocks_in_order(self):
|
||||
raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
assistant_msgs = [m for m in result["messages"] if m["role"] == "assistant"]
|
||||
assert any("manifest" in m["content"].lower() for m in assistant_msgs)
|
||||
# msg-002 has text + tool_use, in that order.
|
||||
assert assistant_msgs
|
||||
types = _block_types(assistant_msgs[0])
|
||||
assert BLOCK_TYPE_TEXT in types
|
||||
assert BLOCK_TYPE_TOOL_USE in types
|
||||
# Order preserved
|
||||
assert types.index(BLOCK_TYPE_TEXT) < types.index(BLOCK_TYPE_TOOL_USE)
|
||||
|
||||
def test_non_text_blocks_skipped_with_warning(self, caplog):
|
||||
import logging
|
||||
def test_tool_use_block_fields(self):
|
||||
raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
with caplog.at_level(logging.WARNING):
|
||||
result = p.normalize_conversation(raw)
|
||||
# The fixture has a tool_use block — should warn
|
||||
|
||||
assistant_msgs = [m for m in result["messages"] if m["role"] == "assistant"]
|
||||
tool_block = _first_block(assistant_msgs[0], BLOCK_TYPE_TOOL_USE)
|
||||
assert tool_block["name"] == "search"
|
||||
assert tool_block["input"] == {"query": "startOS docs"}
|
||||
assert tool_block["tool_id"] == "tool-001"
|
||||
|
||||
def test_image_block_emits_image_placeholder(self):
|
||||
raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
msg004 = [
|
||||
m for m in result["messages"]
|
||||
if any(b.get("type") == BLOCK_TYPE_IMAGE_PLACEHOLDER for b in (m.get("blocks") or []))
|
||||
]
|
||||
assert msg004
|
||||
img = _first_block(msg004[0], BLOCK_TYPE_IMAGE_PLACEHOLDER)
|
||||
assert img["ref"] == "claude-image-uuid-1"
|
||||
|
||||
def test_unknown_block_type_records_loss(self):
|
||||
from src.blocks import BLOCK_TYPE_UNKNOWN as _UNK
|
||||
raw = {
|
||||
"uuid": "test-unknown",
|
||||
"name": "T",
|
||||
"chat_messages": [
|
||||
{
|
||||
"uuid": "m1",
|
||||
"sender": "human",
|
||||
"content": [{"type": "future_block_xyz", "data": "..."}],
|
||||
}
|
||||
],
|
||||
}
|
||||
p = self._get_provider()
|
||||
report = LossReport()
|
||||
result = p.normalize_conversation(raw, report)
|
||||
assert any(
|
||||
"tool_use" in r.message or "rich content" in r.message
|
||||
for r in caplog.records
|
||||
b.get("type") == _UNK
|
||||
for m in result["messages"]
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
assert report.unknown_blocks["future_block_xyz"] == 1
|
||||
|
||||
def test_message_count_matches(self):
|
||||
raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
|
||||
def test_thinking_block(self):
|
||||
raw = {
|
||||
"uuid": "thinking-test",
|
||||
"name": "T",
|
||||
"chat_messages": [
|
||||
{
|
||||
"uuid": "m1",
|
||||
"sender": "assistant",
|
||||
"content": [
|
||||
{"type": "thinking", "thinking": "Let me reason about this."},
|
||||
{"type": "text", "text": "Here's the answer."},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
assert result["message_count"] == len(result["messages"])
|
||||
types = _block_types(result["messages"][0])
|
||||
assert BLOCK_TYPE_THINKING in types
|
||||
assert BLOCK_TYPE_TEXT in types
|
||||
|
||||
def test_roles_normalized(self):
|
||||
raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
|
||||
def test_tool_result_with_nested_text_blocks(self):
|
||||
raw = {
|
||||
"uuid": "tool-result-test",
|
||||
"name": "T",
|
||||
"chat_messages": [
|
||||
{
|
||||
"uuid": "m1",
|
||||
"sender": "assistant",
|
||||
"content": [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": "tool-001",
|
||||
"content": [
|
||||
{"type": "text", "text": "search hit 1"},
|
||||
{"type": "text", "text": "search hit 2"},
|
||||
],
|
||||
"is_error": False,
|
||||
}
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
for msg in result["messages"]:
|
||||
assert msg["role"] in ("user", "assistant", "system")
|
||||
tool_result = _first_block(result["messages"][0], BLOCK_TYPE_TOOL_RESULT)
|
||||
assert tool_result is not None
|
||||
assert "search hit 1" in tool_result["output"]
|
||||
assert "search hit 2" in tool_result["output"]
|
||||
assert tool_result["is_error"] is False
|
||||
|
||||
def test_human_sender_maps_to_user(self):
|
||||
raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
|
||||
@@ -207,3 +688,188 @@ class TestClaudeNormalization:
|
||||
roles = {m["role"] for m in result["messages"]}
|
||||
assert "user" in roles
|
||||
assert "human" not in roles
|
||||
|
||||
def test_loss_report_messages_recorded(self):
|
||||
raw = json.loads((FIXTURES / "claude_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
report = LossReport()
|
||||
result = p.normalize_conversation(raw, report)
|
||||
assert report.messages_rendered == len(result["messages"])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# v0.4.1 — execution_output, system_error, tether_browsing_display, conv_id
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestChatGPTToolOutputs:
|
||||
"""v0.4.1 ChatGPT tool-role content_types map onto tool_result blocks."""
|
||||
|
||||
def _get_provider(self):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||
import requests
|
||||
p._session = requests.Session()
|
||||
p._org_id = None
|
||||
p._project_ids = []
|
||||
p._project_map = {}
|
||||
p._project_name_cache = {}
|
||||
return p
|
||||
|
||||
def test_execution_output_emits_tool_result_with_metadata(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
exec_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
b.get("type") == BLOCK_TYPE_TOOL_RESULT
|
||||
and b.get("tool_name") == "container.exec"
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert exec_msgs, "expected execution_output to render as tool_result"
|
||||
block = next(
|
||||
b for b in exec_msgs[0]["blocks"] if b.get("type") == BLOCK_TYPE_TOOL_RESULT
|
||||
)
|
||||
assert block["output"].startswith("Hello from container.exec")
|
||||
assert block["is_error"] is False
|
||||
assert block["summary"] == "Reading skill documentation"
|
||||
|
||||
def test_execution_output_message_role_is_tool(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
tool_msgs = [m for m in result["messages"] if m["role"] == "tool"]
|
||||
assert tool_msgs, "tool-role messages must pass through (filter lifted in v0.4.0)"
|
||||
|
||||
def test_empty_execution_output_skipped(self, caplog):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
with caplog.at_level(logging.DEBUG, logger="src.providers.chatgpt"):
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
# The empty execution_output (author.name="python") must NOT appear.
|
||||
python_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
b.get("type") == BLOCK_TYPE_TOOL_RESULT and b.get("tool_name") == "python"
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert not python_msgs, "empty execution_output should be skipped"
|
||||
assert any("Skipping empty execution_output" in r.message for r in caplog.records)
|
||||
|
||||
def test_system_error_emits_error_tool_result(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
web_err = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
b.get("type") == BLOCK_TYPE_TOOL_RESULT
|
||||
and b.get("tool_name") == "web"
|
||||
and b.get("is_error") is True
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert web_err, "system_error should render as tool_result with is_error=True"
|
||||
block = next(b for b in web_err[0]["blocks"] if b.get("tool_name") == "web")
|
||||
assert "503" in block["output"]
|
||||
|
||||
def test_tether_browsing_display_spinner_skipped(self, caplog):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
with caplog.at_level(logging.DEBUG, logger="src.providers.chatgpt"):
|
||||
result = p.normalize_conversation(raw)
|
||||
|
||||
spinner_msgs = [
|
||||
m for m in result["messages"]
|
||||
if any(
|
||||
b.get("type") == BLOCK_TYPE_TOOL_RESULT and b.get("tool_name") == "file_search"
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
]
|
||||
assert not spinner_msgs, "spinner tether_browsing_display should be skipped"
|
||||
assert any("tether_browsing_display spinner" in r.message for r in caplog.records)
|
||||
|
||||
def test_tether_browsing_display_populated_renders_defensively(self):
|
||||
"""Defensive case (never observed in real data) — populated browse renders."""
|
||||
conv = {
|
||||
"id": "test-tether",
|
||||
"title": "T",
|
||||
"create_time": 1700000000.0,
|
||||
"update_time": 1700000001.0,
|
||||
"mapping": {
|
||||
"root": {"id": "root", "message": None, "parent": None, "children": ["m1"]},
|
||||
"m1": {
|
||||
"id": "m1",
|
||||
"parent": "root",
|
||||
"children": [],
|
||||
"message": {
|
||||
"id": "m1",
|
||||
"author": {"role": "tool", "name": "browser"},
|
||||
"content": {
|
||||
"content_type": "tether_browsing_display",
|
||||
"result": "Found 3 results about kubernetes ingress.",
|
||||
"summary": "ingress search",
|
||||
"assets": None,
|
||||
"tether_id": None,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(conv)
|
||||
assert any(
|
||||
b.get("type") == BLOCK_TYPE_TOOL_RESULT and b.get("tool_name") == "browser"
|
||||
for m in result["messages"]
|
||||
for b in (m.get("blocks") or [])
|
||||
)
|
||||
|
||||
|
||||
class TestChatGPTConvIdFallback:
|
||||
"""v0.4.1: live ChatGPT detail responses use conversation_id, not id."""
|
||||
|
||||
def _get_provider(self):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||
import requests
|
||||
p._session = requests.Session()
|
||||
p._org_id = None
|
||||
p._project_ids = []
|
||||
p._project_map = {}
|
||||
p._project_name_cache = {}
|
||||
return p
|
||||
|
||||
def test_falls_back_to_conversation_id(self):
|
||||
raw = {
|
||||
"conversation_id": "live-chatgpt-uuid",
|
||||
"title": "T",
|
||||
"create_time": 1700000000.0,
|
||||
"update_time": 1700000001.0,
|
||||
"mapping": {
|
||||
"root": {"id": "root", "message": None, "parent": None, "children": []},
|
||||
},
|
||||
}
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
assert result["id"] == "live-chatgpt-uuid"
|
||||
|
||||
def test_id_takes_precedence_when_both_present(self):
|
||||
raw = {
|
||||
"id": "from-id",
|
||||
"conversation_id": "from-conversation-id",
|
||||
"title": "T",
|
||||
"create_time": 1700000000.0,
|
||||
"update_time": 1700000001.0,
|
||||
"mapping": {
|
||||
"root": {"id": "root", "message": None, "parent": None, "children": []},
|
||||
},
|
||||
}
|
||||
p = self._get_provider()
|
||||
result = p.normalize_conversation(raw)
|
||||
assert result["id"] == "from-id"
|
||||
|
||||
+2
-2
@@ -57,13 +57,13 @@ class TestBuildExportPath:
|
||||
path = build_export_path(
|
||||
Path("/exports"), "claude", "my-project", "2024-06-01T00:00:00Z", "file.md"
|
||||
)
|
||||
assert str(path) == "/exports/claude/my-project/2024/file.md"
|
||||
assert str(path) == "/exports/claude/my-project.2024/file.md"
|
||||
|
||||
def test_no_project_uses_no_project_slug(self):
|
||||
path = build_export_path(
|
||||
Path("/exports"), "chatgpt", None, "2024-06-01T00:00:00Z", "file.md"
|
||||
)
|
||||
assert "no-project" in str(path)
|
||||
assert "no-project.2024" in str(path)
|
||||
|
||||
def test_provider_project_structure_omits_year(self):
|
||||
path = build_export_path(
|
||||
|
||||
Reference in New Issue
Block a user