Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 JesseMarkowitzandClaude Opus 5
|
04ebb8987c | ||
|
JesseMarkowitzandClaude Opus 5
|
8c3e00a520 | ||
|
JesseMarkowitzandClaude Opus 5.5
|
805689c649 | ||
|
JesseMarkowitzandClaude Opus 5.5
|
23c6e1f512 | ||
|
JesseMarkowitzandClaude Opus 5.5
|
64068bb19b | ||
|
JesseMarkowitzandClaude Opus 5.5
|
b6ce636891 | ||
|
JesseMarkowitz
|
d3745e1de4 |
+10
-2
@@ -6,10 +6,18 @@
|
||||
|
||||
# --- ChatGPT ---
|
||||
# How to get: open chatgpt.com in Chrome → F12 → Application tab
|
||||
# → Cookies → https://chatgpt.com → find the two cookie chunks:
|
||||
# → Cookies → https://chatgpt.com → find the session token cookie. Chrome splits a
|
||||
# cookie only above ~4KB, so you will see ONE of these two layouts:
|
||||
#
|
||||
# __Secure-next-auth.session-token (the whole value) → CHATGPT_SESSION_TOKEN
|
||||
# (leave _1 empty)
|
||||
# or, when the token was large enough to be split:
|
||||
# __Secure-next-auth.session-token.0 (starts with "eyJ") → CHATGPT_SESSION_TOKEN
|
||||
# __Secure-next-auth.session-token.1 (the remainder) → CHATGPT_SESSION_TOKEN_1
|
||||
# Token type: JWE. Typically valid for ~7 days.
|
||||
#
|
||||
# CHATGPT_SESSION_TOKEN_1 is OPTIONAL — leave it empty when there is no .1 cookie.
|
||||
# But if a .1 cookie does exist, you must copy it: a partial .0 fails silently
|
||||
# (HTTP 200 with no accessToken). Token type: JWE. Typically valid for ~7 days.
|
||||
CHATGPT_SESSION_TOKEN=
|
||||
CHATGPT_SESSION_TOKEN_1=
|
||||
|
||||
|
||||
@@ -24,6 +24,9 @@ exports/
|
||||
!FUTURE.md
|
||||
!FUTURE-ARCHIVE.md
|
||||
!CHANGELOG.md
|
||||
# The split-out documentation (README.md's detail pages). Without this the
|
||||
# blanket *.md above would ignore every new docs/ page silently, with no error.
|
||||
!docs/*.md
|
||||
|
||||
# Cache and logs
|
||||
cache/
|
||||
|
||||
@@ -3,6 +3,93 @@
|
||||
All notable changes to this project will be documented here.
|
||||
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
|
||||
## [1.0.0] - 2026-10-20
|
||||
|
||||
The archive does what it was built to do, on both machines, unattended: four
|
||||
providers, incremental and resumable, Markdown plus Joplin, on a schedule, with
|
||||
results pushed to a phone and failures that announce themselves. What remains on
|
||||
the roadmap is one optional consolidation (`FUTURE.md` §1), not a gap — so this
|
||||
is 1.0.0 rather than another 0.x.
|
||||
|
||||
This release is mostly the unglamorous half of that claim: four bugs that each
|
||||
made a scheduled run lie about its own outcome, and the documentation split that
|
||||
939 lines had outgrown.
|
||||
|
||||
### Fixed
|
||||
- **A transient 5xx on an image download lost the image permanently, and a timeout lost the whole conversation.** An asset download is two hops: `GET /files/{id}/download` mints a signed URL, then that URL is fetched for the bytes. Only the first went through `_make_request`. The second was a bare `session.request` with no 5xx handling at all, which produced two distinct failures in the 2026-10-20 09:00 run:
|
||||
|
||||
```
|
||||
09:04:05 .../files/<id>/download → 500 Retry 1/3 in 2.2s
|
||||
09:04:17 → 500 Retry 2/3 in 4.3s
|
||||
09:04:32 → 500 Retry 3/3 in 9.1s
|
||||
09:04:42 → 200 ← hop 1 recovered on attempt 4
|
||||
09:04:47 WARNING Signed URL returned HTTP 500 ← hop 2, one try, gone
|
||||
```
|
||||
|
||||
Hop 1 was patiently retried and won; hop 2 failed once and gave up. A second asset was lost the same way to a **503**, so this was never specific to 500. Both assets downloaded cleanly on 2026-10-20 at 10:00 — 106 KB and 1.3 MB — confirming the bytes were there all along and only the retry policy lost them.
|
||||
|
||||
Worse, a third asset **timed out**, and that aborted the entire conversation. `curl_cffi`'s `RequestException` subclasses `OSError`, so an un-wrapped timeout on hop 2 slipped past `resolve_media`'s `except ProviderError` and was caught by the export loop's `except OSError`, which reported it as **"File write failed for conversation 6a945155"** — a conversation that had already downloaded ~20 images. Nothing was wrong with any file write. The design intent in `media.py` ("Failures keep placeholders; never fatal") was bypassed because the exception took a path nobody had anticipated.
|
||||
|
||||
The retry loop is now factored out of `_make_request` into `_send_with_retries`, which returns the raw response so a caller wanting **bytes** gets identical handling to one wanting JSON. Hop 2 goes through it, which fixes both bugs at once: 5xx is retried, `Retry-After` is honoured, and a timeout arrives as `ProviderError` — which `resolve_media` already knows how to placeholder. The hop is also now logged, which it never was; its absence from the log is what made this take a debug run to find. `_pace()` is no longer called by hand there, because `_send_with_retries` paces every attempt — pacing twice would have added `REQUEST_DELAY` to all 88 assets in that run.
|
||||
|
||||
Assets get a more patient ladder than the conversation API, since they fail for their own reasons: an immediate retry (free, and a single bad backend node is the commonest shape of a transient 5xx), then ~2/4/8/16s with jitter. **Jitter is deliberately kept** rather than using fixed steps — a storage incident fails many assets at once, and un-jittered retries would hit a recovering backend in lockstep. The ladder is bounded by a wall-clock budget per asset, and the final attempt's read timeout is clamped to what remains, because the timeout case costs 40s a go and six unbounded attempts of it would eat a scheduled run's execution window (two hours, per the Windows task definition).
|
||||
|
||||
### Added
|
||||
- **Transiently failed assets are re-attempted on later runs.** Previously `mark_exported` ran regardless, so a conversation whose image 500'd was "up to date" forever and the image was unrecoverable without hand-editing the manifest. Failures are now split by whether retrying can possibly help: `expired-or-missing` (404) and `forbidden` (403) are permanent — that was settled exhaustively on 2026-08-17 for 19 assets whose bytes OpenAI lost in a Library migration — while 5xx, timeouts and network blips are transient. Transient refs are recorded against the conversation as `pending_media`, which makes `is_cached` treat it as stale so the next run re-fetches them. Re-exporting is cheap: media already on disk is reused and Joplin resource IDs are keyed by path, so only the missing asset is fetched and only the new resource uploaded. Capped at 3 runs (`MAX_MEDIA_RETRY_RUNS`), because an asset that 5xxs every morning would otherwise re-export and re-sync its conversation forever — one of today's conversations took 3m15s and 50 resource uploads. The refs are kept after the cap as a record of what was lost.
|
||||
|
||||
This leaves the "per-conversation cache reset" item in `FUTURE.md` correctly closed: the case that would have motivated it — recovering one conversation after a transient asset failure — now heals itself, and the documented workaround (edit the manifest) still covers the rest.
|
||||
- **A media circuit breaker, so a bad day at the provider costs one quiet morning.** Five consecutive *transient* asset failures stop downloads for the remainder of the run; the skipped assets are recorded as retryable rather than abandoned, so the next run collects them. Permanent failures never count toward tripping it — a deleted upload says nothing about the provider's health, and this archive holds 19 of those that would otherwise trip it on a perfectly healthy server. Any success resets the streak, so ordinary scattered losses never trip it either. Assets already on disk are still linked while the breaker is open, since that needs no network.
|
||||
- **Media failures appear in the push notification.** `format_summary` now adds `N media failed` per provider. The 2026-10-20 run lost two assets and pushed a clean "AI archive OK" with no hint of it — the counts existed only in the log file, which is somewhere you have to think to go and look. Counts only, never asset ids, for the same reason conversation titles are excluded: a topic on public ntfy.sh is readable by anyone who knows its name. Media failures deliberately do **not** raise the notification's priority or mark the run failed — the transcript is archived and a transient asset recovers by itself.
|
||||
|
||||
### Changed
|
||||
- **The README is split into a short overview plus four documents in `docs/`,** closing `FUTURE.md` §2. It had reached 939 lines / 6,490 words / 48 KB / 96 headings — a ~25-minute read doing four unrelated jobs at once, and still growing (it was 830 / 5,562 / 37 KB / 61 when the split was first proposed). `README.md` is now 303 lines and keeps only what you need to install and configure the tool: the ToS warning, installation, `doctor`, `.env` reference, output structure, security notes. The rest moved verbatim — no section was rewritten or dropped, and every original heading still exists, so every anchor still resolves *somewhere*:
|
||||
|
||||
| Document | Content |
|
||||
|----------|---------|
|
||||
| `docs/providers.md` | Session tokens, lifetimes and expiry detection, the `auth` wizard, ChatGPT Projects, Claude Code sessions, Codex sessions |
|
||||
| `docs/cli.md` | CLI reference — all eleven commands |
|
||||
| `docs/scheduling.md` | systemd timer, Windows task, what to know before relying on it, notifications |
|
||||
| `docs/troubleshooting.md` | How the cache works, every failure message, media download failures, filing a bug report |
|
||||
|
||||
The cost this deferral was protecting against is real and is now paid: a bookmark or note pointing at a README anchor that moved (`#cli-reference`, `#troubleshooting`, `#getting-your-session-tokens`, …) no longer resolves, because the anchor is in another file. The three questions `FUTURE.md` left open are settled as follows. **`.gitignore` did need the fix it predicted** — the blanket `*.md` rule (exported conversations are Markdown and may be private) matched all four new pages, and `git check-ignore` confirmed every one would have been committed-never and ignored silently; `!docs/*.md` is added alongside the existing per-file re-includes, with a comment saying why. **Cross-file anchors are used, not avoided**, but only for the four links that already existed between these sections, each verified to point at a heading that still exists. **Gitea rendering of relative `docs/` links is unverified** — it cannot be checked before pushing, so confirm it after.
|
||||
|
||||
### Notes
|
||||
- Verified live rather than only in tests: the two assets lost in the morning run were re-downloaded successfully through the new path (`doctor` all green, 414 tests passing, a read-only `export --dry-run --provider chatgpt` exercising the refactored request core across 4 listing pages / 379 conversations). The cross-run retry and the breaker are covered by unit tests only — reproducing a multi-day storage incident against the live provider is not something this change waited for.
|
||||
|
||||
- **Every scheduled Claude Code sync since 2026-09-24 died with `RecursionError`.** Claude Code's `fork` subagents write a transcript that opens with a copy of the parent turn that spawned them — the fork's own `Agent` call included. `_extract_messages` folds a subagent inline whenever it meets its spawn call, so it met that copy inside the fork, folded the same fork again, and recursed until Python's limit. One fork anywhere in `~/.claude/projects` was enough to fail the whole provider; seven existed across three sessions, and the Codex half of the run, unaffected, hid the cause behind a generic exit 1.
|
||||
|
||||
The recursive pass now carries the spawn ids being expanded around it and skips a tool_use whose id is among them — the enclosing subagent block already stands for that call. Because the set accumulates, a longer cycle (A spawns B, whose transcript re-spawns A) stops too. The fork's `<fork-boilerplate>` preamble — generic worker rules the harness prepends — is now stripped with the other harness tags, so a fork block opens with its actual directive.
|
||||
|
||||
`TestSubagentFold.test_fork_containing_its_own_spawn_call` reproduces the on-disk shape (a `fork-context-ref` record, the copied spawn call, the boilerplate-wrapped directive) and fails with the original `RecursionError` against the unfixed code. Verified against the real archive: all 85 sessions normalize, the seven forks each render as one subagent block.
|
||||
|
||||
- **A scheduled run that crashed sent no notification, and the providers that survived said "OK".** `sync` pushes its ntfy result from the end of a run it finished, so the `RecursionError` above — and any crash, any exit before the sync starts (the ToS gate, a cache error), a launcher that cannot build its venv — sent nothing at all. Worse, the unit runs each provider as its own `sync`, so codex kept pushing a low-priority "AI archive OK" every morning for the eleven days claude-code was dead: a broken provider was indistinguishable from a quiet day.
|
||||
|
||||
The systemd unit's `ExecStart` is now `scheduling/run-sync.sh`, which carries the old per-provider loop and pushes a high-priority **FAILED** notification for any run that exits non-zero without the app's "Sync completed with failures" banner (printed right after the app's own push, so an app-reported failure is not reported twice). The push names the provider and, for a crash, the exception class only — `claude-code: crashed (RecursionError)` — never its message, holding to `src/notify.py`'s counts-only rule for a topic anyone can read. It honours `NTFY_NOTIFY=off` and reads `NTFY_*` the way the app does: environment first, then `.env`. Re-run `install-systemd-timer.sh` to pick it up.
|
||||
|
||||
Verified against a fake launcher and a local capture server: a crash pushes with the class and without the message, an exit before the sync pushes, an app-reported failure and a success push nothing extra, `off` pushes nothing, and the run still exits 1 if any provider failed.
|
||||
|
||||
The Windows task gets the same backstop. Its action is now `scheduling\run-sync.ps1` (via `powershell.exe -File`, window hidden) in place of the inline `cmd.exe` loop, with the same rules: every provider attempted, exit 1 if any failed, a FAILED push carrying the class only for a run that died unreported, `NTFY_NOTIFY=off` honoured, `NTFY_*` read from the environment first. Because Task Scheduler keeps no output, each run's output is appended to `cache\logs\scheduled-sync.log` (one rollover to `.1` past 1 MB), and the push points there. The wrapper sets `PYTHONIOENCODING=utf-8`: with stdout redirected to a file rather than a console, Python would otherwise encode in the ANSI code page and fail on the first `✓`. Re-run `Register-AiChatSyncTask.ps1` with your providers to pick it up.
|
||||
|
||||
Verified on Linux under PowerShell 7.6 with a stand-in `cmd.exe`, the same fake launcher and capture server: the same five outcomes as above, plus a bearer token, an environment `NTFY_TOPIC` overriding `.env`, an unreachable server (warning, exit still 1) and the `chatgpt,claude` single-argument form. Since confirmed on winmini under Windows PowerShell 5.1 and Task Scheduler (2026-10-20): `Register-AiChatSyncTask.ps1 -Provider chatgpt,claude` registered the wrapper, and a `Start-ScheduledTask` run of it completed with `LastTaskResult: 0`, writing both providers' output to `cache\logs\scheduled-sync.log`. Registering is a one-time write, so a `git pull` does **not** update an already-registered task — re-run the registration script after taking this change. The one path still unexercised on Windows is the FAILED push itself: it needs a run that dies without reporting, which has not happened there yet.
|
||||
|
||||
- **An expired Claude session key reported a raw JSON dump instead of how to fix it.** `_make_request` routed only **401** to the auth handler (`src/providers/base.py`), and claude.ai does not use 401 — an invalid or expired `sessionKey` comes back as `403 permission_error` with `details.error_code = account_session_invalid`. So the one message that names the cookie, its ~30-day lifetime and the DevTools path to refresh it could never fire for Claude. What the user got instead was the generic 4xx path: `HTTP 403 — error: {'type': 'permission_error', 'message': 'Invalid authorization'…}`, which reads like a permissions problem with the account and not like "your key expired, here is how to replace it."
|
||||
|
||||
Measured live 2026-09-20 against `GET /api/organizations`: a valid key returns 200, while an expired key, a deliberately malformed key and **no cookie at all** return byte-identical 403s carrying that code — i.e. the API treats a dead session as an absent one. This is the same mistake as the ChatGPT media 403s below: assuming 403 means "forbidden" when the service uses it for "unauthenticated."
|
||||
|
||||
Auth detection is now a provider decision rather than a hardcoded status. `BaseProvider._is_auth_failure(response)` defaults to 401 and `ClaudeProvider` overrides it to add 403 **matched on `account_session_invalid`**, not on the bare status — so a genuine permission error, which carries a different code, is still reported as itself rather than being mislabelled an expired key. `_handle_401` is renamed `_handle_auth_failure` and takes the response, because a handler named for one status that must handle two is how this stayed hidden; its messages now state the status actually observed instead of asserting "401 Unauthorized". ChatGPT is unaffected: it does not override the default, so the deleted-asset 403 path is untouched.
|
||||
|
||||
Seven regression tests cover the split (`TestAuthFailureDetection`), including the two that matter most: a Claude 403 with a *different* error code must **not** be treated as an auth failure, and a ChatGPT 403 must not either. The docs that repeated the wrong premise — `README.md`'s expiry table and "When Tokens Expire" section, the `auth` wizard's on-screen note, and the `ClaudeProvider` docstring — are corrected in the same change.
|
||||
|
||||
- **The docs claimed both ChatGPT cookie chunks were required; they are not.** `README.md` stated flatly that "ChatGPT splits large session tokens across two cookies to stay under the browser's 4KB cookie limit. Both are required," and `.env.example` documented only the chunked layout — so a machine whose session token happens to fit in a single `__Secure-next-auth.session-token` cookie looked broken, with the user hunting for a `.1` that does not exist. Chrome splits a cookie only above ~4KB, so the layout varies by session size and the *same account* can be chunked on one machine and not on another.
|
||||
|
||||
The code was already correct: `CHATGPT_SESSION_TOKEN_1` is optional (`src/providers/chatgpt.py:162`) and the `auth` wizard already told you to paste a lone cookie into `.0` and leave `.1` blank. Only the reference docs were wrong, and they are the ones read when setting up a new machine.
|
||||
|
||||
Measured 2026-09-20 against `/api/auth/session`, reassembling a real 4089-byte token to test each naming: chunked `.0`+`.1` → 200 with an `accessToken`; the whole value under the unchunked name → 200 with an `accessToken`; the whole value under `.0` alone → 200 with an `accessToken`. The server reassembles a complete value sent under `.0`, so both layouts authenticate as the code already assumed. A *partial* `.0` with its `.1` omitted is the one combination that fails, and it fails **silently** — HTTP 200 with no `accessToken` rather than an error — which is now documented in both files alongside the correction.
|
||||
|
||||
- **The test suite sent real push notifications to the developer's phone.** `TestSyncCommand` invokes the actual `sync` command, which calls `load_config()`, which calls `load_dotenv()` — so the real `.env` was loaded and its live `NTFY_TOPIC` used for the POST. Every `pytest` run fired three or four pushes, including a fabricated "codex: 3 conversation(s) failed to export" straight out of a fixture, which is worse than noise: it reports a failure that never happened. Nothing appeared in `cache/logs/exporter.log` to explain it, because every test invocation passes `--no-log-file`.
|
||||
|
||||
A `tests/conftest.py` autouse fixture now neutralises the environment for every test: `NTFY_TOPIC`/`NTFY_TOKEN` are emptied, and `NTFY_SERVER` and `JOPLIN_API_URL` are pointed at a closed local port, so a stray topic cannot reach the internet and a test cannot write notes into a real Joplin instance. The values are **emptied rather than deleted** — `load_dotenv(override=False)` skips only keys already present, so deleting one lets `.env` put it back. Verified by instrumenting `requests` across a full run: zero outbound requests, where the same instrumentation without the fixture records POSTs to the live ntfy topic.
|
||||
|
||||
## [0.9.0] - 2026-08-18
|
||||
|
||||
### Fixed
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Planned Future Work
|
||||
|
||||
> **Status 2026-08-18 (v0.9.0).** The tool archives four providers — two web
|
||||
> (`chatgpt`, `claude`) and two local agent-transcript (`claude-code`, `codex`)
|
||||
> — on a schedule, on Linux and Windows, reporting results by push notification.
|
||||
> Two items below are genuinely planned. Everything else has shipped or been
|
||||
> **Status 2026-10-20 (v1.0.0).** The tool archives four providers — two web (`chatgpt`,
|
||||
> `claude`) and two local agent-transcript (`claude-code`, `codex`) — on a
|
||||
> schedule, on Linux and Windows, reporting results by push notification.
|
||||
> **One item below is genuinely planned.** Everything else has shipped or been
|
||||
> decided against.
|
||||
|
||||
Completed work moves to the changelog; this file holds only what is *not* built
|
||||
@@ -71,42 +71,6 @@ What this needs, and what it does *not*:
|
||||
Meanwhile Joplin is sufficient — it already syncs (encrypted) offsite, and it is
|
||||
where the archive is actually read. This is a "nice eventually", not a gap.
|
||||
|
||||
## 2. Split the README into separate documents
|
||||
|
||||
The README is 830 lines / 5,562 words / 37 KB — about a 25-minute read, with 61
|
||||
headings. An H2-only table of contents was added 2026-08-18 and helps, but it
|
||||
treats the symptom: the file is doing at least four unrelated jobs at once.
|
||||
|
||||
Rough shape of a split:
|
||||
|
||||
| Document | Content today |
|
||||
|----------|---------------|
|
||||
| `README.md` | What it is, install, first run, a pointer to the rest |
|
||||
| `docs/providers.md` | ChatGPT Projects, Claude Code Sessions, Codex Sessions, session tokens |
|
||||
| `docs/cli.md` | CLI Reference — ~250 lines on its own, the single biggest section |
|
||||
| `docs/scheduling.md` | Scheduling a Daily Run, notifications |
|
||||
| `docs/troubleshooting.md` | Troubleshooting, How the Cache Works |
|
||||
|
||||
Not urgent, because it has a real cost the TOC did not: any existing link into a
|
||||
README section — a bookmark, a note, another repo, a commit message — breaks
|
||||
when that section moves to another file. Worth folding into the next substantial
|
||||
README edit rather than doing as a change of its own.
|
||||
|
||||
Two things to settle when it happens:
|
||||
|
||||
- **`.gitignore` ignores `*.md` on purpose** — exported conversations are
|
||||
Markdown and may contain private content — and re-includes each doc by name
|
||||
(`!README.md`, `!FUTURE.md`, …). New files under `docs/` will be silently
|
||||
ignored, with no error, until `!docs/*.md` is added. This already bit the
|
||||
creation of `FUTURE-ARCHIVE.md` on 2026-08-18.
|
||||
- Whether `docs/` renders acceptably on the Gitea instance hosting this repo.
|
||||
Relative links between Markdown files do work there, but confirm before
|
||||
splitting rather than after.
|
||||
- Whether anchors in the split files are stable enough to link *between*
|
||||
documents, or whether cross-references should point at file tops only. GFM
|
||||
anchors derive from heading text and break silently on a reword — the same
|
||||
fragility the TOC already carries.
|
||||
|
||||
---
|
||||
|
||||
# Decided against
|
||||
@@ -142,3 +106,4 @@ Detail for each release is in `CHANGELOG.md`.
|
||||
- **v0.7.0** — `canary` drift detection; real ChatGPT token-health check on `doctor`; removal of the non-viable browser-cookie path
|
||||
- **v0.8.0** — Claude Code coverage reopened: subagent capture as folded `<details>`, repo `[tags]` in titles, its own `AI-ClaudeCode` notebook with self-healing note moves, multi-root scanning (`CLAUDE_CODE_DIR`, `CLAUDE_CONFIG_DIR`)
|
||||
- **v0.9.0** — Codex CLI provider; launcher scripts removing the virtualenv ceremony on both platforms; `sync` with a meaningful exit code; daily scheduling for Linux and Windows; ntfy push notifications; ChatGPT project attribution via `gizmo_id` and the `projects` command; a silent data-loss fix in both local providers (`splitlines` breaking on U+0085/U+2028/U+2029)
|
||||
- **v1.0.0** — Reliability pass on unattended runs: asset downloads retried on 5xx with a per-asset time budget, transiently failed assets recovered on later runs, a media circuit breaker, media failures surfaced in the push notification; a timeout on an asset no longer aborts its conversation; claude.ai's 403 session-invalid recognised as an expired key; `fork` subagents no longer recurse forever; a crashed scheduled run pushes a FAILED notification on both platforms; the README split into an overview plus `docs/`
|
||||
|
||||
@@ -8,23 +8,26 @@ Supports incremental sync — only new or updated conversations are exported on
|
||||
|
||||
## Contents
|
||||
|
||||
**This page** — what it is, installing it, the first run, configuration and where files land.
|
||||
|
||||
- [Terms of Service Warning](#terms-of-service-warning)
|
||||
- [Installation](#installation)
|
||||
- [First Run: Run Doctor](#first-run-run-doctor)
|
||||
- [Getting Your Session Tokens](#getting-your-session-tokens)
|
||||
- [The `auth` Command](#the-auth-command)
|
||||
- [`.env` Setup](#env-setup)
|
||||
- [ChatGPT Projects](#chatgpt-projects)
|
||||
- [Claude Code Sessions](#claude-code-sessions)
|
||||
- [Codex Sessions](#codex-sessions)
|
||||
- [Scheduling a Daily Run](#scheduling-a-daily-run)
|
||||
- [Output Structure](#output-structure)
|
||||
- [CLI Reference](#cli-reference)
|
||||
- [How the Cache Works](#how-the-cache-works)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Documentation](#documentation)
|
||||
- [Future Work](#future-work)
|
||||
- [Security Notes](#security-notes)
|
||||
|
||||
**Detail lives in `docs/`:**
|
||||
|
||||
| Document | Covers |
|
||||
|----------|--------|
|
||||
| [docs/providers.md](docs/providers.md) | Session tokens and how to refresh them, the `auth` wizard, ChatGPT Projects, Claude Code sessions, Codex sessions |
|
||||
| [docs/cli.md](docs/cli.md) | CLI reference — every command and flag |
|
||||
| [docs/scheduling.md](docs/scheduling.md) | The daily run on Linux and Windows, and push notifications |
|
||||
| [docs/troubleshooting.md](docs/troubleshooting.md) | How the cache works, what each failure means, filing a bug report |
|
||||
|
||||
---
|
||||
|
||||
## Terms of Service Warning
|
||||
@@ -151,63 +154,6 @@ It probes one conversation per provider and checks only the fields the parser de
|
||||
|
||||
---
|
||||
|
||||
## Getting Your Session Tokens
|
||||
|
||||
Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
|
||||
|
||||
Tokens are entered manually — copy them from your browser's DevTools and run the wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
The wizard detects your OS, shows the correct DevTools shortcut, and writes the values to `.env` without echoing them to the terminal.
|
||||
|
||||
> **Why no auto-extraction from the browser?** Modern Chromium browsers (Chrome 127+, current Brave) encrypt cookies on Windows with **App-Bound Encryption**: the key is bound to the browser through a SYSTEM-level service, so reading cookies off disk requires Administrator rights and a PsExec-style SYSTEM impersonation that antivirus flags as credential theft — and it still fails on Brave specifically. There is no reliable, non-invasive way to pull these cookies automatically, so the manual DevTools flow below is the supported path.
|
||||
|
||||
### Token Lifetimes
|
||||
|
||||
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
||||
|----------|-------------|----------|-----------------|
|
||||
| ChatGPT | `__Secure-next-auth.session-token.0` + `.1` | refresh ~weekly | `error` field of `/api/auth/session` — `doctor` reports "ChatGPT token active". The token is an encrypted JWE, so its `exp` is **not** readable client-side, and the `expires` field is a misleading rolling window; the `error` (`RefreshAccessTokenError` when dead) is the honest signal. |
|
||||
| Claude | `sessionKey` | ~30 days | Opaque token — only detectable via 401 response |
|
||||
|
||||
### Finding Tokens in Chrome DevTools
|
||||
|
||||
1. Open the provider's website and make sure you're logged in
|
||||
2. Press **F12** (Windows/Linux) or **Cmd+Option+I** (macOS) to open DevTools
|
||||
3. Click the **Application** tab
|
||||
4. In the left panel, expand **Cookies** and click the site URL
|
||||
5. Find the cookie by name and copy its **Value**
|
||||
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find **two** cookies:
|
||||
- `__Secure-next-auth.session-token.0` — copy Value (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`
|
||||
- `__Secure-next-auth.session-token.1` — copy Value → `CHATGPT_SESSION_TOKEN_1`
|
||||
|
||||
ChatGPT splits large session tokens across two cookies to stay under the browser's 4KB cookie limit. Both are required.
|
||||
|
||||
**Claude:** go to `https://claude.ai` → find `sessionKey` → copy Value
|
||||
|
||||
### When Tokens Expire
|
||||
|
||||
When a token expires you'll see a `401 Unauthorized` error. To refresh:
|
||||
- Re-run the `auth` wizard: `ai-chat-exporter auth`
|
||||
- Or manually update the value in your `.env` file
|
||||
|
||||
---
|
||||
|
||||
## The `auth` Command
|
||||
|
||||
The easiest way to configure tokens is the interactive wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
This walks you through finding your token, validates it, shows the expiry date (ChatGPT only), and offers to write it to your `.env` automatically. Tokens are never echoed to the terminal.
|
||||
|
||||
---
|
||||
|
||||
## `.env` Setup
|
||||
|
||||
Copy `.env.example` to `.env` and fill in your values:
|
||||
@@ -255,7 +201,7 @@ cp .env.example .env
|
||||
|
||||
A topic on public ntfy.sh is readable by anyone who knows its name, so
|
||||
notifications carry per-provider counts and a machine name only — never
|
||||
conversation titles. See [Getting notified](#getting-notified).
|
||||
conversation titles. See [Getting notified](docs/scheduling.md#getting-notified).
|
||||
|
||||
### Cache & logging
|
||||
|
||||
@@ -267,193 +213,6 @@ conversation titles. See [Getting notified](#getting-notified).
|
||||
|
||||
---
|
||||
|
||||
## ChatGPT Projects
|
||||
|
||||
ChatGPT project conversations are stored separately from your main conversation list and require extra configuration.
|
||||
|
||||
### Finding your project IDs
|
||||
|
||||
The quickest way is to let the exporter find them:
|
||||
|
||||
```
|
||||
ai-chat-exporter projects
|
||||
```
|
||||
|
||||
It lists every project your conversations actually belong to, marks which are
|
||||
missing from `.env`, and prints a paste-ready `CHATGPT_PROJECT_IDS=` line
|
||||
(`--write` updates `.env` for you). If your ChatGPT account does not include
|
||||
the project on conversation summaries, add `--deep` and it reads each
|
||||
conversation's detail instead — slower, one request per conversation, but
|
||||
complete.
|
||||
|
||||
Why it matters: project attribution is resolved from each conversation's own
|
||||
`gizmo_id`, so exports file correctly whether or not a project is configured.
|
||||
But the *listing* pass still needs `CHATGPT_PROJECT_IDS` — conversations that
|
||||
live only inside a project never appear in the default conversation list, so an
|
||||
unlisted project's chats are never fetched at all. Every export run also names
|
||||
any unconfigured project it encounters.
|
||||
|
||||
To find them by hand instead:
|
||||
|
||||
1. Open ChatGPT and click a Project in the left sidebar
|
||||
2. Look at the browser URL — it will look like:
|
||||
`https://chatgpt.com/g/g-p-68c2b2b3037c8191890036fb4ae3ed9f-my-project/project`
|
||||
3. Copy the `g-p-…` part (everything up to but not including the slug after the second `-`)
|
||||
|
||||
Add all your project IDs to `.env` as a comma-separated list:
|
||||
|
||||
```
|
||||
CHATGPT_PROJECT_IDS=g-p-68c2b2b3037c8191890036fb4ae3ed9f,g-p-anotherprojectid
|
||||
```
|
||||
|
||||
The `auth` wizard can also guide you through this step interactively.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Sessions
|
||||
|
||||
The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider claude-code
|
||||
ai-chat-exporter joplin --provider claude-code
|
||||
```
|
||||
|
||||
Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Sessions become notes under their own top-level **`AI-ClaudeCode`** Joplin notebook, in a sub-notebook per launch folder. The provider is included in `--provider all` whenever a sessions directory exists.
|
||||
|
||||
**Subagents.** Claude Code stores subagent (Task-tool) transcripts as separate files under `<session>/subagents/`; each is folded into its parent session inline, as a collapsible `<details>` block labeled with the subagent's type and description (its own tool traffic is collapsed like the main dialogue).
|
||||
|
||||
**Repo tags.** Sessions launched from a workspace root all share one folder-named notebook, so titles carry the repos each session touched — `Resume StartWRT project work [start-technologies]` — for at-a-glance scanning and search. A file's repo is the git repository it lives in (nearest ancestor with a `.git`), resolved from the tool paths in the transcript, so work is tagged wherever it happened — even across workspaces — and config/one-off files are ignored (they aren't repos). Repos are frequency-ordered and capped at 3. To never tag specific repos, set `CLAUDE_CODE_REPO_TAG_IGNORE` (comma-separated). Note this reads your current git layout, so a repo you later delete or move drops from the tag on re-export.
|
||||
|
||||
**Multiple locations.** By default the provider scans `~/.claude/projects/` plus `$CLAUDE_CONFIG_DIR/projects` when `CLAUDE_CONFIG_DIR` is set. To scan additional roots (e.g. other machines' sessions copied onto this box), set `CLAUDE_CODE_DIR` to a `:`-separated list:
|
||||
|
||||
```bash
|
||||
CLAUDE_CODE_DIR="$HOME/.claude/projects:/mnt/backup/laptop/.claude/projects"
|
||||
```
|
||||
|
||||
Sessions from all roots are merged by folder (no per-machine label); if the same session UUID appears in two roots, the newer copy wins. Note the exporter only sees this machine's disk and cannot recover sessions Claude Code has already pruned — run it regularly.
|
||||
|
||||
---
|
||||
|
||||
## Codex Sessions
|
||||
|
||||
The `codex` provider archives your local [Codex CLI](https://chatgpt.com/codex) agent transcripts — same deal as Claude Code: no tokens, no API, no ToS exposure. Rollout files are read from `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider codex
|
||||
ai-chat-exporter joplin --provider codex
|
||||
```
|
||||
|
||||
Exports are prose-only by default, with the same placeholder format as Claude Code, into their own top-level **`AI-Codex`** Joplin notebook. Repo tags work the same way (`CODEX_REPO_TAG_IGNORE` to suppress), and additional roots can be scanned with `CODEX_DIR` (`:`-separated); `CODEX_HOME`'s `sessions/` is picked up automatically when that variable is set.
|
||||
|
||||
Three things differ from Claude Code, all forced by how Codex stores its data:
|
||||
|
||||
**Reasoning cannot be exported.** Codex encrypts it at rest — every reasoning record carries `encrypted_content` with no plaintext summary in any layer of the file. It is always dropped and counted; `EXPORTER_HIDDEN_CONTENT=full` cannot bring it back.
|
||||
|
||||
**Incomplete tool calls are reported.** Codex writes each session twice in one file: its own typed items (what actually ran) and the raw model-facing wire format (everything attempted). The provider reads the typed layer — it is already decoded, and it omits harness plumbing that would otherwise need stripping — but cross-checks the raw layer for calls that never produced a result, so placeholders read `3 calls: exec_command ×3 (+2 did not complete)`. Those are commands that failed to launch, that you aborted, or that were still running when the turn ended.
|
||||
|
||||
**Cloud tasks are out of scope.** `codex cloud` tasks run server-side and are reachable at `chatgpt.com/backend-api/api/codex/tasks`, but local CLI sessions are never uploaded there, so the cloud API is not an alternative source for these transcripts and this provider stays entirely offline. If you start using `codex cloud exec`, those transcripts would be cloud-only and would need separate work.
|
||||
|
||||
---
|
||||
|
||||
## Scheduling a Daily Run
|
||||
|
||||
`sync` chains `export` then `joplin` in one invocation, which is what a scheduler
|
||||
wants — one command, and a meaningful exit code so a failed run is visible
|
||||
instead of silent.
|
||||
|
||||
```bash
|
||||
./ai-chat-exporter sync # every configured provider
|
||||
./ai-chat-exporter sync --provider codex # just one
|
||||
```
|
||||
|
||||
### Linux (systemd user timer)
|
||||
|
||||
```bash
|
||||
./scheduling/install-systemd-timer.sh --provider claude-code --provider codex
|
||||
./scheduling/install-systemd-timer.sh --uninstall
|
||||
```
|
||||
|
||||
Defaults to 09:00 daily (`--time 21:30` to change). `Persistent=true` means a
|
||||
machine that was off at the scheduled time runs the archive at next boot rather
|
||||
than skipping the day. To archive while logged out, `loginctl enable-linger $USER`.
|
||||
|
||||
Check on it with `systemctl --user list-timers aichat-sync.timer` and
|
||||
`journalctl --user -u aichat-sync.service -n 50`.
|
||||
|
||||
### Windows (Task Scheduler)
|
||||
|
||||
```powershell
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Provider chatgpt,claude
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Unregister
|
||||
```
|
||||
|
||||
Per-user task, no admin rights needed. `-StartWhenAvailable` is the counterpart
|
||||
of systemd's `Persistent=true`.
|
||||
|
||||
### What to know before you rely on it
|
||||
|
||||
**List only the providers that work unattended on that machine.** The local
|
||||
providers (`claude-code`, `codex`) need no credentials and always work. The web
|
||||
providers depend on a session token that expires and can only be refreshed by
|
||||
hand via DevTools — so on a machine where that token is stale, scheduling them
|
||||
means a failed run every single day, which is a good way to learn to ignore
|
||||
failures you actually want to see. That is why `--provider` is repeatable in both
|
||||
installers: schedule the coding machine for `claude-code` + `codex`, the browser
|
||||
machine for `chatgpt` + `claude`.
|
||||
|
||||
**Both installers pass `--joplin-optional`.** If Joplin desktop isn't running,
|
||||
the sync warns instead of failing: the export has already captured the local
|
||||
transcripts (the part that can disappear), and the notes are rebuilt from the
|
||||
cache on the next run that finds Joplin up.
|
||||
|
||||
**One provider failing does not skip the rest.** Both installers loop over the
|
||||
providers in a single action rather than one action each, because systemd
|
||||
`oneshot` stops at the first failing `ExecStart` and Task Scheduler reports only
|
||||
the last action's result. Every provider is attempted; the run still exits
|
||||
non-zero if any failed.
|
||||
|
||||
### Getting notified
|
||||
|
||||
A scheduled run is silent by default. Output goes to three pull-only places: the
|
||||
exporter's own log (`cache/logs/exporter.log`), the systemd journal on Linux
|
||||
(`journalctl --user -u aichat-sync.service`), and Task Scheduler's
|
||||
`LastTaskResult` on Windows.
|
||||
|
||||
To have runs report back, set an [ntfy](https://ntfy.sh) topic in `.env`:
|
||||
|
||||
```bash
|
||||
NTFY_TOPIC=my-archive-topic
|
||||
```
|
||||
|
||||
Then check it works before waiting on a scheduled run:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter notify # show settings
|
||||
ai-chat-exporter notify --test # send a test push
|
||||
```
|
||||
|
||||
`sync` pushes a result whenever a topic is configured — a success carries the
|
||||
per-provider counts at low/default priority, a failure carries the reason at
|
||||
high priority with an alert tag, so a failed archive is distinguishable from a
|
||||
quiet one on your phone. `NTFY_NOTIFY=failure` notifies only on failure; `off`
|
||||
disables it; `--notify` / `--no-notify` override per run.
|
||||
|
||||
The message includes the **machine name**, which matters because both machines
|
||||
archive into one topic. It contains counts only — never conversation titles. A
|
||||
topic on public ntfy.sh is readable by anyone who knows its name, so if you want
|
||||
it private, self-host (`NTFY_SERVER`) or use an access-controlled topic with
|
||||
`NTFY_TOKEN`.
|
||||
|
||||
A notification is never fatal: if ntfy is unreachable, the run logs a warning and
|
||||
still reports its real exit code.
|
||||
|
||||
**Acknowledge the ToS notice once, interactively.** It's stored in the cache
|
||||
manifest per machine. Until then a scheduled run exits 1 with an explanation
|
||||
rather than hanging on a prompt no one can answer.
|
||||
|
||||
---
|
||||
|
||||
## Output Structure
|
||||
|
||||
All exported files go under `EXPORT_DIR`. The folder structure maps directly to Joplin notebooks.
|
||||
@@ -503,344 +262,34 @@ Each provider+project combination maps to a flat Joplin notebook created automat
|
||||
|
||||
---
|
||||
|
||||
## CLI Reference
|
||||
|
||||
### Global flags
|
||||
|
||||
```
|
||||
--verbose / -v DEBUG output to console
|
||||
--quiet / -q WARNING and above only
|
||||
--debug DEBUG + full tracebacks + redacted API response bodies
|
||||
--no-log-file Disable file logging
|
||||
--version Print version and exit
|
||||
```
|
||||
|
||||
### `auth` — Interactive token setup
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth # manual wizard (DevTools flow)
|
||||
```
|
||||
|
||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut, then writes the values to `.env`.
|
||||
|
||||
### `doctor` — Health check
|
||||
|
||||
```bash
|
||||
ai-chat-exporter doctor
|
||||
```
|
||||
|
||||
Checks: token presence, JWT validity and expiry, directory permissions, disk space, live API reachability. Exits with code 0 if all pass, 1 if any fail.
|
||||
|
||||
### `export` — Export conversations
|
||||
|
||||
```bash
|
||||
# Export everything (new/updated only)
|
||||
ai-chat-exporter export
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter export --provider claude
|
||||
|
||||
# JSON output
|
||||
ai-chat-exporter export --format json
|
||||
|
||||
# Both Markdown and JSON
|
||||
ai-chat-exporter export --format both
|
||||
|
||||
# Only conversations updated since a date
|
||||
ai-chat-exporter export --since 2024-06-01
|
||||
|
||||
# Only conversations in a specific project (case-insensitive substring)
|
||||
ai-chat-exporter export --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter export --project none
|
||||
|
||||
# Write to a custom directory
|
||||
ai-chat-exporter export --output /path/to/my/notes
|
||||
|
||||
# Preview without writing anything
|
||||
ai-chat-exporter export --dry-run
|
||||
```
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--force`, `--dry-run`
|
||||
|
||||
**Re-rendering the whole archive after an upgrade.** New formatting or features (collapse policy, media downloads) only change conversations as they're re-exported. To re-render everything you already have, use `--force` — it re-exports every conversation even if unchanged, **without** `cache --clear`, so your Joplin note links are preserved (a later `joplin` run updates the existing notes instead of duplicating them).
|
||||
|
||||
A force re-render runs as a tracked "campaign": each run re-renders the least-recently-exported conversations, the "still to go" count shrinks each run, and finished providers do no further work. The simplest approach is one uncapped pass:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force # re-renders everything in one paced pass
|
||||
ai-chat-exporter joplin # update notes + upload media
|
||||
```
|
||||
|
||||
Or spread the load with `--max-conversations`, re-running until it reports "Force re-render complete":
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force --max-conversations 50 # repeat until complete
|
||||
```
|
||||
|
||||
### `list` — List conversations
|
||||
|
||||
```bash
|
||||
# List all conversations for all providers
|
||||
ai-chat-exporter list
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter list --provider chatgpt
|
||||
|
||||
# Filter by project
|
||||
ai-chat-exporter list --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter list --project none
|
||||
```
|
||||
|
||||
Fetches and displays all conversations without exporting them. Useful for verifying what the tool can see before running an export.
|
||||
|
||||
### `joplin` — Sync to Joplin
|
||||
|
||||
```bash
|
||||
# Sync all pending conversations to Joplin
|
||||
ai-chat-exporter joplin
|
||||
|
||||
# Preview what would be synced without sending anything
|
||||
ai-chat-exporter joplin --dry-run
|
||||
|
||||
# Sync a single provider
|
||||
ai-chat-exporter joplin --provider chatgpt
|
||||
|
||||
# Sync only conversations in a specific project
|
||||
ai-chat-exporter joplin --project "learning python"
|
||||
|
||||
# Sync only conversations outside any project
|
||||
ai-chat-exporter joplin --project none
|
||||
```
|
||||
|
||||
Reads the local export cache and pushes each exported Markdown file to Joplin as a note. Notebooks are created automatically. Re-running is safe — notes are updated (not duplicated).
|
||||
|
||||
**Prerequisites:**
|
||||
1. Run `export` first to generate the Markdown files
|
||||
2. Open Joplin → Tools → Options → Web Clipper → enable the service
|
||||
3. Copy the Authorization token and add `JOPLIN_API_TOKEN=<token>` to your `.env`
|
||||
4. Joplin desktop must be open when you run this command
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--project NAME`, `--dry-run`
|
||||
|
||||
### `sync` — Export and sync in one run
|
||||
|
||||
```bash
|
||||
# The whole archive run: export, then push to Joplin
|
||||
ai-chat-exporter sync
|
||||
|
||||
# One provider
|
||||
ai-chat-exporter sync --provider codex
|
||||
|
||||
# Export only; don't touch Joplin
|
||||
ai-chat-exporter sync --skip-joplin
|
||||
|
||||
# Joplin being closed is a warning, not a failure (used by the schedulers)
|
||||
ai-chat-exporter sync --joplin-optional
|
||||
```
|
||||
|
||||
Equivalent to `export` followed by `joplin` with the same `--provider`. Intended
|
||||
for scheduled runs — see [Scheduling a Daily Run](#scheduling-a-daily-run).
|
||||
|
||||
Unlike the individual commands, `sync` sets a **meaningful exit code**: non-zero
|
||||
if any conversation failed to export or any note failed to sync. A provider whose
|
||||
listing call fails outright (an expired web session token being the usual cause)
|
||||
counts its whole batch as failed. A provider that is simply unconfigured, or that
|
||||
had nothing new, is ordinary success. `export` on its own always exits 0, which
|
||||
is fine when you're reading the summary table and useless to a scheduler.
|
||||
|
||||
`--joplin-optional` downgrades an unreachable Joplin to a warning: the export has
|
||||
already captured the local transcripts, and the notes are rebuilt from the cache
|
||||
by the next run that finds Joplin open.
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--since YYYY-MM-DD`, `--hidden-content [full|placeholder|omit]`, `--max-conversations N`, `--skip-joplin`, `--joplin-optional`, `--notify/--no-notify`, `--dry-run`
|
||||
|
||||
Note this is a deliberate subset of `export`'s options — `--format`, `--output`,
|
||||
`--project`, `--download-media` and `--force` are not passed through. Use
|
||||
`export` directly for those. (`--download-media` still applies from `.env`; the
|
||||
flag is only a per-run override.)
|
||||
|
||||
### `notify` — Push-notification settings and test
|
||||
|
||||
```bash
|
||||
# Show the current settings
|
||||
ai-chat-exporter notify
|
||||
|
||||
# Send a test push to confirm the topic works
|
||||
ai-chat-exporter notify --test
|
||||
```
|
||||
|
||||
Shows the resolved ntfy configuration and which machine name will appear in the
|
||||
title. See [Getting notified](#getting-notified) for what a scheduled run sends.
|
||||
|
||||
Options: `--test`
|
||||
|
||||
### `prune` — Delete stale export files
|
||||
|
||||
```bash
|
||||
# Preview what would be deleted
|
||||
ai-chat-exporter prune --dry-run
|
||||
|
||||
# Delete (asks for confirmation; -y skips the prompt)
|
||||
ai-chat-exporter prune
|
||||
```
|
||||
|
||||
Deletes export files no longer referenced by the cache manifest — leftovers
|
||||
from old folder layouts or fixed bugs that would otherwise double-sync into
|
||||
Joplin. Refuses to run when the manifest is empty (e.g. right after
|
||||
`cache --clear`) so it can never wipe a freshly cleared archive. The `doctor`
|
||||
command separately verifies that every manifest entry's file exists on disk.
|
||||
|
||||
### `projects` — Discover ChatGPT project IDs
|
||||
|
||||
```bash
|
||||
# List the projects your conversations belong to
|
||||
ai-chat-exporter projects
|
||||
|
||||
# Also inspect conversations whose listing entry doesn't name a project
|
||||
ai-chat-exporter projects --deep
|
||||
|
||||
# Write the discovered IDs straight into .env
|
||||
ai-chat-exporter projects --write
|
||||
```
|
||||
|
||||
`CHATGPT_PROJECT_IDS` is maintained by hand, and a project missing from it is
|
||||
invisible to the listing pass — conversations that live *only* inside that
|
||||
project are never fetched at all. This reports every project your conversations
|
||||
belong to, marks the ones absent from `.env`, and prints a paste-ready line.
|
||||
|
||||
`--deep` fetches each conversation's detail when the listing doesn't name its
|
||||
project: complete, but one request per conversation, so it's slow.
|
||||
|
||||
Options: `--deep`, `--write`
|
||||
|
||||
### `canary` — Check for provider API drift
|
||||
|
||||
```bash
|
||||
ai-chat-exporter canary
|
||||
ai-chat-exporter canary --provider chatgpt
|
||||
```
|
||||
|
||||
The web providers are undocumented internal APIs that can change shape without
|
||||
notice, and the failure mode is silent — a renamed field means content is
|
||||
quietly dropped rather than an error being raised. The canary fetches one
|
||||
listing page and one conversation per provider and asserts only the fields the
|
||||
normalizer actually depends on.
|
||||
|
||||
Findings are `ERROR` (a load-bearing field is missing or mistyped — the parser
|
||||
will break or silently lose data) or `WARN` (something unfamiliar appeared;
|
||||
worth investigating, not necessarily broken). **Exits non-zero on any ERROR**,
|
||||
so it can be scheduled or run in CI. Local providers have no remote schema and
|
||||
are not probed.
|
||||
|
||||
Options: `--provider [chatgpt|claude|all]`
|
||||
|
||||
### `cache` — Manage the sync manifest
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
ai-chat-exporter cache --show
|
||||
|
||||
# Clear all cached entries (forces full re-export next run)
|
||||
ai-chat-exporter cache --clear
|
||||
|
||||
# Clear a single provider
|
||||
ai-chat-exporter cache --clear --provider claude
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How the Cache Works
|
||||
|
||||
The cache manifest lives at `cache/manifest.json` (inside the install directory) and records every exported conversation: its title, project, `updated_at` timestamp, output file path, and (after Joplin sync) the Joplin note ID.
|
||||
|
||||
On every `export` run:
|
||||
1. Fetch the full conversation list from the provider
|
||||
2. Compare each conversation's `updated_at` against the manifest
|
||||
3. Export only conversations that are new or have been updated
|
||||
4. Write each successfully exported conversation to the manifest **immediately** (not batched)
|
||||
|
||||
On every `joplin` run:
|
||||
1. Read the manifest to find conversations not yet synced to Joplin, or re-exported since last sync
|
||||
2. Push each pending Markdown file to Joplin (create or update)
|
||||
3. Store the Joplin note ID in the manifest so subsequent runs update rather than duplicate
|
||||
|
||||
**This design makes every run inherently resumable.** If the tool is interrupted for any reason — rate limit, network drop, Ctrl+C, crash — simply re-run the same command. It will skip already-processed conversations and continue from where it stopped.
|
||||
|
||||
To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `401 Unauthorized`
|
||||
Your session token has expired.
|
||||
- Run `ai-chat-exporter auth` to get a new token interactively
|
||||
- Or manually copy a fresh cookie value into your `.env` file
|
||||
|
||||
Note: Claude's `sessionKey` is an opaque string — the only way to know it's expired is the 401 error. ChatGPT JWTs have an `exp` claim that the `doctor` command can decode and display.
|
||||
|
||||
### `429 Rate Limited`
|
||||
The tool automatically pauses, saves progress, and exits with a clear message showing how many conversations were exported vs remaining. Just re-run the same export command to resume — the cache picks up exactly where it left off.
|
||||
|
||||
### Joplin: "JOPLIN_API_TOKEN is not set"
|
||||
You need to configure the token before running the `joplin` command:
|
||||
1. Open Joplin desktop
|
||||
2. Go to Tools → Options → Web Clipper
|
||||
3. Enable the Web Clipper service
|
||||
4. Copy the Authorization token shown on that page
|
||||
5. Add `JOPLIN_API_TOKEN=<token>` to your `.env` file
|
||||
|
||||
### Joplin: "Joplin is not responding"
|
||||
Joplin desktop must be running when you run the `joplin` command. The Web Clipper service shuts down when Joplin is closed.
|
||||
|
||||
### Joplin: "Joplin rejected the API token (HTTP 401)"
|
||||
The token in `JOPLIN_API_TOKEN` doesn't match what Joplin expects. Get a fresh token from Joplin → Tools → Options → Web Clipper → Authorization token.
|
||||
|
||||
### Joplin: note timed out
|
||||
If you see a timeout error, Joplin took longer than `JOPLIN_REQUEST_TIMEOUT` seconds (default: 30) to respond. Possible causes:
|
||||
- The conversation is very large and Joplin is slow to index it
|
||||
- Joplin is busy syncing or loading a large library
|
||||
- Joplin has frozen — try restarting it
|
||||
|
||||
To increase the timeout: add `JOPLIN_REQUEST_TIMEOUT=60` to your `.env`.
|
||||
|
||||
### ChatGPT project conversations not appearing
|
||||
Make sure you've added the project IDs to `CHATGPT_PROJECT_IDS` in your `.env`. See [ChatGPT Projects](#chatgpt-projects) for how to find them. Project conversations are not included in the default conversation listing — they must be fetched separately.
|
||||
|
||||
### Schema warnings in logs (`Unexpected API response shape`)
|
||||
The provider's internal API may have changed. Run with `--debug`, sanitize the output (remove any personal content), and check the project's GitHub Issues for known fixes.
|
||||
|
||||
### Non-text content warnings
|
||||
Since v0.4.0, rich content is preserved as typed blocks in the export. ChatGPT voice transcripts render as text and audio assets as `📎 File attached` placeholders with size and duration metadata. Anything the extractor doesn't recognise renders as a visible `> ⚠️ Unsupported content` block naming the type and observed keys, *and* increments a counter in the post-export summary so you can tell whether real content is being silently skipped.
|
||||
|
||||
### Embedded images and media
|
||||
Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as ``. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Assets that have expired on the provider's side (older AI-generated images often do) keep their text placeholder and are tallied as `media failed` in the run summary — they're never fatal to the export. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
|
||||
|
||||
### Tool output / hidden context collapsed
|
||||
Since v0.6.0, content that was invisible in the ChatGPT web UI is collapsed by default to one-line placeholders like `> 🔧 Tool output — file_search (15.0 KB) — omitted`. This is ChatGPT's file-retrieval tool re-injecting your attached files on every run — it can be 90% of a conversation's bytes while containing none of the dialogue. Custom Instructions collapse the same way (`> ℹ️ Hidden context`). The post-export summary lists every collapsed message by origin with the total KB omitted. To keep everything, set `EXPORTER_HIDDEN_CONTENT=full` in `.env` or pass `--hidden-content full`.
|
||||
|
||||
### Empty export / all conversations skipped
|
||||
No new or updated conversations since your last run. To verify: `ai-chat-exporter cache --show`. To force a full re-export: `ai-chat-exporter cache --clear`.
|
||||
|
||||
### Filing a bug report
|
||||
1. Run with `--debug`: `ai-chat-exporter export --debug 2>&1 | tee debug.log`
|
||||
2. Remove any personal conversation content from `debug.log`
|
||||
3. Open a GitHub Issue with the sanitized log and the exact command you ran
|
||||
## Documentation
|
||||
|
||||
This page covers installing and configuring the exporter. Everything else is in
|
||||
`docs/`:
|
||||
|
||||
- **[Providers and Tokens](docs/providers.md)** — getting a session token out of
|
||||
DevTools, what each one's lifetime is and how expiry is detected, the `auth`
|
||||
wizard, and the setup each provider needs: ChatGPT Projects (which are not in
|
||||
the default conversation listing), Claude Code sessions, Codex sessions.
|
||||
- **[CLI Reference](docs/cli.md)** — `auth`, `doctor`, `export`, `list`,
|
||||
`joplin`, `sync`, `notify`, `prune`, `projects`, `canary`, `cache`, with the
|
||||
flags and examples for each.
|
||||
- **[Running on a Schedule](docs/scheduling.md)** — the systemd user timer, the
|
||||
Windows scheduled task, what to know before relying on either, and how run
|
||||
results reach your phone.
|
||||
- **[Troubleshooting](docs/troubleshooting.md)** — how the cache decides what to
|
||||
re-export, every error message the tool produces and what it means, media
|
||||
download failures, and what to include in a bug report.
|
||||
|
||||
---
|
||||
|
||||
## Future Work
|
||||
|
||||
See `FUTURE.md` for the full roadmap. Current priorities:
|
||||
See `FUTURE.md` for the full roadmap. Current priority:
|
||||
|
||||
- **A StartOS service** that centralises every machine's conversations into one
|
||||
corpus and owns the Joplin connection, so each machine only has to upload
|
||||
(`FUTURE.md` §8)
|
||||
- **Splitting this README** into a short overview plus separate documents
|
||||
(`FUTURE.md` §1)
|
||||
|
||||
---
|
||||
|
||||
|
||||
+254
@@ -0,0 +1,254 @@
|
||||
# CLI Reference
|
||||
|
||||
Every command, its flags and what it does.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [Scheduling](scheduling.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Global flags
|
||||
|
||||
```
|
||||
--verbose / -v DEBUG output to console
|
||||
--quiet / -q WARNING and above only
|
||||
--debug DEBUG + full tracebacks + redacted API response bodies
|
||||
--no-log-file Disable file logging
|
||||
--version Print version and exit
|
||||
```
|
||||
|
||||
## `auth` — Interactive token setup
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth # manual wizard (DevTools flow)
|
||||
```
|
||||
|
||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut, then writes the values to `.env`.
|
||||
|
||||
## `doctor` — Health check
|
||||
|
||||
```bash
|
||||
ai-chat-exporter doctor
|
||||
```
|
||||
|
||||
Checks: token presence, JWT validity and expiry, directory permissions, disk space, live API reachability. Exits with code 0 if all pass, 1 if any fail.
|
||||
|
||||
## `export` — Export conversations
|
||||
|
||||
```bash
|
||||
# Export everything (new/updated only)
|
||||
ai-chat-exporter export
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter export --provider claude
|
||||
|
||||
# JSON output
|
||||
ai-chat-exporter export --format json
|
||||
|
||||
# Both Markdown and JSON
|
||||
ai-chat-exporter export --format both
|
||||
|
||||
# Only conversations updated since a date
|
||||
ai-chat-exporter export --since 2024-06-01
|
||||
|
||||
# Only conversations in a specific project (case-insensitive substring)
|
||||
ai-chat-exporter export --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter export --project none
|
||||
|
||||
# Write to a custom directory
|
||||
ai-chat-exporter export --output /path/to/my/notes
|
||||
|
||||
# Preview without writing anything
|
||||
ai-chat-exporter export --dry-run
|
||||
```
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--force`, `--dry-run`
|
||||
|
||||
**Re-rendering the whole archive after an upgrade.** New formatting or features (collapse policy, media downloads) only change conversations as they're re-exported. To re-render everything you already have, use `--force` — it re-exports every conversation even if unchanged, **without** `cache --clear`, so your Joplin note links are preserved (a later `joplin` run updates the existing notes instead of duplicating them).
|
||||
|
||||
A force re-render runs as a tracked "campaign": each run re-renders the least-recently-exported conversations, the "still to go" count shrinks each run, and finished providers do no further work. The simplest approach is one uncapped pass:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force # re-renders everything in one paced pass
|
||||
ai-chat-exporter joplin # update notes + upload media
|
||||
```
|
||||
|
||||
Or spread the load with `--max-conversations`, re-running until it reports "Force re-render complete":
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force --max-conversations 50 # repeat until complete
|
||||
```
|
||||
|
||||
## `list` — List conversations
|
||||
|
||||
```bash
|
||||
# List all conversations for all providers
|
||||
ai-chat-exporter list
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter list --provider chatgpt
|
||||
|
||||
# Filter by project
|
||||
ai-chat-exporter list --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter list --project none
|
||||
```
|
||||
|
||||
Fetches and displays all conversations without exporting them. Useful for verifying what the tool can see before running an export.
|
||||
|
||||
## `joplin` — Sync to Joplin
|
||||
|
||||
```bash
|
||||
# Sync all pending conversations to Joplin
|
||||
ai-chat-exporter joplin
|
||||
|
||||
# Preview what would be synced without sending anything
|
||||
ai-chat-exporter joplin --dry-run
|
||||
|
||||
# Sync a single provider
|
||||
ai-chat-exporter joplin --provider chatgpt
|
||||
|
||||
# Sync only conversations in a specific project
|
||||
ai-chat-exporter joplin --project "learning python"
|
||||
|
||||
# Sync only conversations outside any project
|
||||
ai-chat-exporter joplin --project none
|
||||
```
|
||||
|
||||
Reads the local export cache and pushes each exported Markdown file to Joplin as a note. Notebooks are created automatically. Re-running is safe — notes are updated (not duplicated).
|
||||
|
||||
**Prerequisites:**
|
||||
1. Run `export` first to generate the Markdown files
|
||||
2. Open Joplin → Tools → Options → Web Clipper → enable the service
|
||||
3. Copy the Authorization token and add `JOPLIN_API_TOKEN=<token>` to your `.env`
|
||||
4. Joplin desktop must be open when you run this command
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--project NAME`, `--dry-run`
|
||||
|
||||
## `sync` — Export and sync in one run
|
||||
|
||||
```bash
|
||||
# The whole archive run: export, then push to Joplin
|
||||
ai-chat-exporter sync
|
||||
|
||||
# One provider
|
||||
ai-chat-exporter sync --provider codex
|
||||
|
||||
# Export only; don't touch Joplin
|
||||
ai-chat-exporter sync --skip-joplin
|
||||
|
||||
# Joplin being closed is a warning, not a failure (used by the schedulers)
|
||||
ai-chat-exporter sync --joplin-optional
|
||||
```
|
||||
|
||||
Equivalent to `export` followed by `joplin` with the same `--provider`. Intended
|
||||
for scheduled runs — see [Scheduling a Daily Run](scheduling.md#scheduling-a-daily-run).
|
||||
|
||||
Unlike the individual commands, `sync` sets a **meaningful exit code**: non-zero
|
||||
if any conversation failed to export or any note failed to sync. A provider whose
|
||||
listing call fails outright (an expired web session token being the usual cause)
|
||||
counts its whole batch as failed. A provider that is simply unconfigured, or that
|
||||
had nothing new, is ordinary success. `export` on its own always exits 0, which
|
||||
is fine when you're reading the summary table and useless to a scheduler.
|
||||
|
||||
`--joplin-optional` downgrades an unreachable Joplin to a warning: the export has
|
||||
already captured the local transcripts, and the notes are rebuilt from the cache
|
||||
by the next run that finds Joplin open.
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--since YYYY-MM-DD`, `--hidden-content [full|placeholder|omit]`, `--max-conversations N`, `--skip-joplin`, `--joplin-optional`, `--notify/--no-notify`, `--dry-run`
|
||||
|
||||
Note this is a deliberate subset of `export`'s options — `--format`, `--output`,
|
||||
`--project`, `--download-media` and `--force` are not passed through. Use
|
||||
`export` directly for those. (`--download-media` still applies from `.env`; the
|
||||
flag is only a per-run override.)
|
||||
|
||||
## `notify` — Push-notification settings and test
|
||||
|
||||
```bash
|
||||
# Show the current settings
|
||||
ai-chat-exporter notify
|
||||
|
||||
# Send a test push to confirm the topic works
|
||||
ai-chat-exporter notify --test
|
||||
```
|
||||
|
||||
Shows the resolved ntfy configuration and which machine name will appear in the
|
||||
title. See [Getting notified](scheduling.md#getting-notified) for what a scheduled run sends.
|
||||
|
||||
Options: `--test`
|
||||
|
||||
## `prune` — Delete stale export files
|
||||
|
||||
```bash
|
||||
# Preview what would be deleted
|
||||
ai-chat-exporter prune --dry-run
|
||||
|
||||
# Delete (asks for confirmation; -y skips the prompt)
|
||||
ai-chat-exporter prune
|
||||
```
|
||||
|
||||
Deletes export files no longer referenced by the cache manifest — leftovers
|
||||
from old folder layouts or fixed bugs that would otherwise double-sync into
|
||||
Joplin. Refuses to run when the manifest is empty (e.g. right after
|
||||
`cache --clear`) so it can never wipe a freshly cleared archive. The `doctor`
|
||||
command separately verifies that every manifest entry's file exists on disk.
|
||||
|
||||
## `projects` — Discover ChatGPT project IDs
|
||||
|
||||
```bash
|
||||
# List the projects your conversations belong to
|
||||
ai-chat-exporter projects
|
||||
|
||||
# Also inspect conversations whose listing entry doesn't name a project
|
||||
ai-chat-exporter projects --deep
|
||||
|
||||
# Write the discovered IDs straight into .env
|
||||
ai-chat-exporter projects --write
|
||||
```
|
||||
|
||||
`CHATGPT_PROJECT_IDS` is maintained by hand, and a project missing from it is
|
||||
invisible to the listing pass — conversations that live *only* inside that
|
||||
project are never fetched at all. This reports every project your conversations
|
||||
belong to, marks the ones absent from `.env`, and prints a paste-ready line.
|
||||
|
||||
`--deep` fetches each conversation's detail when the listing doesn't name its
|
||||
project: complete, but one request per conversation, so it's slow.
|
||||
|
||||
Options: `--deep`, `--write`
|
||||
|
||||
## `canary` — Check for provider API drift
|
||||
|
||||
```bash
|
||||
ai-chat-exporter canary
|
||||
ai-chat-exporter canary --provider chatgpt
|
||||
```
|
||||
|
||||
The web providers are undocumented internal APIs that can change shape without
|
||||
notice, and the failure mode is silent — a renamed field means content is
|
||||
quietly dropped rather than an error being raised. The canary fetches one
|
||||
listing page and one conversation per provider and asserts only the fields the
|
||||
normalizer actually depends on.
|
||||
|
||||
Findings are `ERROR` (a load-bearing field is missing or mistyped — the parser
|
||||
will break or silently lose data) or `WARN` (something unfamiliar appeared;
|
||||
worth investigating, not necessarily broken). **Exits non-zero on any ERROR**,
|
||||
so it can be scheduled or run in CI. Local providers have no remote schema and
|
||||
are not probed.
|
||||
|
||||
Options: `--provider [chatgpt|claude|all]`
|
||||
|
||||
## `cache` — Manage the sync manifest
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
ai-chat-exporter cache --show
|
||||
|
||||
# Clear all cached entries (forces full re-export next run)
|
||||
ai-chat-exporter cache --clear
|
||||
|
||||
# Clear a single provider
|
||||
ai-chat-exporter cache --clear --provider claude
|
||||
```
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
# Providers and Tokens
|
||||
|
||||
How to obtain and refresh the session tokens for the two web providers, and the
|
||||
per-provider setup for ChatGPT Projects, Claude Code and Codex.
|
||||
|
||||
← [README](../README.md) · [Configuration](../README.md#env-setup) · [CLI Reference](cli.md) · [Scheduling](scheduling.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Getting Your Session Tokens
|
||||
|
||||
Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
|
||||
|
||||
Tokens are entered manually — copy them from your browser's DevTools and run the wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
The wizard detects your OS, shows the correct DevTools shortcut, and writes the values to `.env` without echoing them to the terminal.
|
||||
|
||||
> **Why no auto-extraction from the browser?** Modern Chromium browsers (Chrome 127+, current Brave) encrypt cookies on Windows with **App-Bound Encryption**: the key is bound to the browser through a SYSTEM-level service, so reading cookies off disk requires Administrator rights and a PsExec-style SYSTEM impersonation that antivirus flags as credential theft — and it still fails on Brave specifically. There is no reliable, non-invasive way to pull these cookies automatically, so the manual DevTools flow below is the supported path.
|
||||
|
||||
### Token Lifetimes
|
||||
|
||||
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
||||
|----------|-------------|----------|-----------------|
|
||||
| ChatGPT | `__Secure-next-auth.session-token` (split into `.0` + `.1` when over ~4KB) | refresh ~weekly | `error` field of `/api/auth/session` — `doctor` reports "ChatGPT token active". The token is an encrypted JWE, so its `exp` is **not** readable client-side, and the `expires` field is a misleading rolling window; the `error` (`RefreshAccessTokenError` when dead) is the honest signal. |
|
||||
| Claude | `sessionKey` | ~30 days | Opaque token — only detectable from an API rejection. claude.ai answers an invalid session with **403** `permission_error` / `account_session_invalid`, **not** 401; `doctor` reports it on the "Claude API reachable" row. |
|
||||
|
||||
### Finding Tokens in Chrome DevTools
|
||||
|
||||
1. Open the provider's website and make sure you're logged in
|
||||
2. Press **F12** (Windows/Linux) or **Cmd+Option+I** (macOS) to open DevTools
|
||||
3. Click the **Application** tab
|
||||
4. In the left panel, expand **Cookies** and click the site URL
|
||||
5. Find the cookie by name and copy its **Value**
|
||||
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find the session token cookie. You will
|
||||
see **one of two layouts**, depending on how large your session token is:
|
||||
|
||||
- **One cookie**, `__Secure-next-auth.session-token` — copy Value → `CHATGPT_SESSION_TOKEN`, and leave `CHATGPT_SESSION_TOKEN_1` empty.
|
||||
- **Two cookies**, `__Secure-next-auth.session-token.0` and `.1` — copy `.0` (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`, and `.1` → `CHATGPT_SESSION_TOKEN_1`.
|
||||
|
||||
Chrome splits a cookie only when it exceeds ~4KB, so a larger session is chunked
|
||||
and a smaller one is not — the same account can differ from machine to machine.
|
||||
`CHATGPT_SESSION_TOKEN_1` is optional; both layouts authenticate, because the
|
||||
server reassembles a complete value sent under the `.0` name.
|
||||
|
||||
What does *not* work is sending a **partial** chunk — `.0` on its own when a `.1`
|
||||
exists. That fails silently: `/api/auth/session` answers HTTP 200 with no
|
||||
`accessToken` rather than an error. If you see two cookies, copy both.
|
||||
|
||||
**Claude:** go to `https://claude.ai` → find `sessionKey` → copy Value
|
||||
|
||||
### When Tokens Expire
|
||||
|
||||
An expired token shows up as an authentication error naming the cookie to
|
||||
refresh and how. The status differs by provider — ChatGPT reports 401, while
|
||||
claude.ai reports **403 "Invalid authorization"** (`account_session_invalid`) —
|
||||
so don't read a 403 from Claude as a permissions problem with your account.
|
||||
|
||||
To refresh:
|
||||
- Re-run the `auth` wizard: `ai-chat-exporter auth`
|
||||
- Or manually update the value in your `.env` file
|
||||
|
||||
`ai-chat-exporter doctor` is the quickest check: the "token set" rows only test
|
||||
that a value is present, so an expired credential passes those and fails on the
|
||||
"API reachable" row.
|
||||
|
||||
---
|
||||
|
||||
## The `auth` Command
|
||||
|
||||
The easiest way to configure tokens is the interactive wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
This walks you through finding your token, validates it, shows the expiry date (ChatGPT only), and offers to write it to your `.env` automatically. Tokens are never echoed to the terminal.
|
||||
|
||||
---
|
||||
|
||||
## ChatGPT Projects
|
||||
|
||||
ChatGPT project conversations are stored separately from your main conversation list and require extra configuration.
|
||||
|
||||
### Finding your project IDs
|
||||
|
||||
The quickest way is to let the exporter find them:
|
||||
|
||||
```
|
||||
ai-chat-exporter projects
|
||||
```
|
||||
|
||||
It lists every project your conversations actually belong to, marks which are
|
||||
missing from `.env`, and prints a paste-ready `CHATGPT_PROJECT_IDS=` line
|
||||
(`--write` updates `.env` for you). If your ChatGPT account does not include
|
||||
the project on conversation summaries, add `--deep` and it reads each
|
||||
conversation's detail instead — slower, one request per conversation, but
|
||||
complete.
|
||||
|
||||
Why it matters: project attribution is resolved from each conversation's own
|
||||
`gizmo_id`, so exports file correctly whether or not a project is configured.
|
||||
But the *listing* pass still needs `CHATGPT_PROJECT_IDS` — conversations that
|
||||
live only inside a project never appear in the default conversation list, so an
|
||||
unlisted project's chats are never fetched at all. Every export run also names
|
||||
any unconfigured project it encounters.
|
||||
|
||||
To find them by hand instead:
|
||||
|
||||
1. Open ChatGPT and click a Project in the left sidebar
|
||||
2. Look at the browser URL — it will look like:
|
||||
`https://chatgpt.com/g/g-p-68c2b2b3037c8191890036fb4ae3ed9f-my-project/project`
|
||||
3. Copy the `g-p-…` part (everything up to but not including the slug after the second `-`)
|
||||
|
||||
Add all your project IDs to `.env` as a comma-separated list:
|
||||
|
||||
```
|
||||
CHATGPT_PROJECT_IDS=g-p-68c2b2b3037c8191890036fb4ae3ed9f,g-p-anotherprojectid
|
||||
```
|
||||
|
||||
The `auth` wizard can also guide you through this step interactively.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Sessions
|
||||
|
||||
The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider claude-code
|
||||
ai-chat-exporter joplin --provider claude-code
|
||||
```
|
||||
|
||||
Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Sessions become notes under their own top-level **`AI-ClaudeCode`** Joplin notebook, in a sub-notebook per launch folder. The provider is included in `--provider all` whenever a sessions directory exists.
|
||||
|
||||
**Subagents.** Claude Code stores subagent (Task-tool) transcripts as separate files under `<session>/subagents/`; each is folded into its parent session inline, as a collapsible `<details>` block labeled with the subagent's type and description (its own tool traffic is collapsed like the main dialogue).
|
||||
|
||||
**Repo tags.** Sessions launched from a workspace root all share one folder-named notebook, so titles carry the repos each session touched — `Resume StartWRT project work [start-technologies]` — for at-a-glance scanning and search. A file's repo is the git repository it lives in (nearest ancestor with a `.git`), resolved from the tool paths in the transcript, so work is tagged wherever it happened — even across workspaces — and config/one-off files are ignored (they aren't repos). Repos are frequency-ordered and capped at 3. To never tag specific repos, set `CLAUDE_CODE_REPO_TAG_IGNORE` (comma-separated). Note this reads your current git layout, so a repo you later delete or move drops from the tag on re-export.
|
||||
|
||||
**Multiple locations.** By default the provider scans `~/.claude/projects/` plus `$CLAUDE_CONFIG_DIR/projects` when `CLAUDE_CONFIG_DIR` is set. To scan additional roots (e.g. other machines' sessions copied onto this box), set `CLAUDE_CODE_DIR` to a `:`-separated list:
|
||||
|
||||
```bash
|
||||
CLAUDE_CODE_DIR="$HOME/.claude/projects:/mnt/backup/laptop/.claude/projects"
|
||||
```
|
||||
|
||||
Sessions from all roots are merged by folder (no per-machine label); if the same session UUID appears in two roots, the newer copy wins. Note the exporter only sees this machine's disk and cannot recover sessions Claude Code has already pruned — run it regularly.
|
||||
|
||||
---
|
||||
|
||||
## Codex Sessions
|
||||
|
||||
The `codex` provider archives your local [Codex CLI](https://chatgpt.com/codex) agent transcripts — same deal as Claude Code: no tokens, no API, no ToS exposure. Rollout files are read from `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider codex
|
||||
ai-chat-exporter joplin --provider codex
|
||||
```
|
||||
|
||||
Exports are prose-only by default, with the same placeholder format as Claude Code, into their own top-level **`AI-Codex`** Joplin notebook. Repo tags work the same way (`CODEX_REPO_TAG_IGNORE` to suppress), and additional roots can be scanned with `CODEX_DIR` (`:`-separated); `CODEX_HOME`'s `sessions/` is picked up automatically when that variable is set.
|
||||
|
||||
Three things differ from Claude Code, all forced by how Codex stores its data:
|
||||
|
||||
**Reasoning cannot be exported.** Codex encrypts it at rest — every reasoning record carries `encrypted_content` with no plaintext summary in any layer of the file. It is always dropped and counted; `EXPORTER_HIDDEN_CONTENT=full` cannot bring it back.
|
||||
|
||||
**Incomplete tool calls are reported.** Codex writes each session twice in one file: its own typed items (what actually ran) and the raw model-facing wire format (everything attempted). The provider reads the typed layer — it is already decoded, and it omits harness plumbing that would otherwise need stripping — but cross-checks the raw layer for calls that never produced a result, so placeholders read `3 calls: exec_command ×3 (+2 did not complete)`. Those are commands that failed to launch, that you aborted, or that were still running when the turn ended.
|
||||
|
||||
**Cloud tasks are out of scope.** `codex cloud` tasks run server-side and are reachable at `chatgpt.com/backend-api/api/codex/tasks`, but local CLI sessions are never uploaded there, so the cloud API is not an alternative source for these transcripts and this provider stays entirely offline. If you start using `codex cloud exec`, those transcripts would be cloud-only and would need separate work.
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
# Running on a Schedule
|
||||
|
||||
Installing the daily run on Linux and Windows, what to know before relying on
|
||||
it, and how results reach your phone.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [CLI Reference](cli.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Scheduling a Daily Run
|
||||
|
||||
`sync` chains `export` then `joplin` in one invocation, which is what a scheduler
|
||||
wants — one command, and a meaningful exit code so a failed run is visible
|
||||
instead of silent.
|
||||
|
||||
```bash
|
||||
./ai-chat-exporter sync # every configured provider
|
||||
./ai-chat-exporter sync --provider codex # just one
|
||||
```
|
||||
|
||||
### Linux (systemd user timer)
|
||||
|
||||
```bash
|
||||
./scheduling/install-systemd-timer.sh --provider claude-code --provider codex
|
||||
./scheduling/install-systemd-timer.sh --uninstall
|
||||
```
|
||||
|
||||
Defaults to 09:00 daily (`--time 21:30` to change). `Persistent=true` means a
|
||||
machine that was off at the scheduled time runs the archive at next boot rather
|
||||
than skipping the day. To archive while logged out, `loginctl enable-linger $USER`.
|
||||
|
||||
Check on it with `systemctl --user list-timers aichat-sync.timer` and
|
||||
`journalctl --user -u aichat-sync.service -n 50`.
|
||||
|
||||
### Windows (Task Scheduler)
|
||||
|
||||
```powershell
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Provider chatgpt,claude
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Unregister
|
||||
```
|
||||
|
||||
Per-user task, no admin rights needed. `-StartWhenAvailable` is the counterpart
|
||||
of systemd's `Persistent=true`. The task runs `scheduling\run-sync.ps1` with its
|
||||
window hidden; Task Scheduler keeps no output, so each run's output is appended
|
||||
to `cache\logs\scheduled-sync.log` (rolled over to `.1` past 1 MB).
|
||||
|
||||
Check on it with `Get-ScheduledTaskInfo -TaskName AiChatExporterSync` (look at
|
||||
`LastTaskResult`: 0 is success) and
|
||||
`Get-Content cache\logs\scheduled-sync.log -Tail 50`.
|
||||
|
||||
### What to know before you rely on it
|
||||
|
||||
**List only the providers that work unattended on that machine.** The local
|
||||
providers (`claude-code`, `codex`) need no credentials and always work. The web
|
||||
providers depend on a session token that expires and can only be refreshed by
|
||||
hand via DevTools — so on a machine where that token is stale, scheduling them
|
||||
means a failed run every single day, which is a good way to learn to ignore
|
||||
failures you actually want to see. That is why `--provider` is repeatable in both
|
||||
installers: schedule the coding machine for `claude-code` + `codex`, the browser
|
||||
machine for `chatgpt` + `claude`.
|
||||
|
||||
**Both installers pass `--joplin-optional`.** If Joplin desktop isn't running,
|
||||
the sync warns instead of failing: the export has already captured the local
|
||||
transcripts (the part that can disappear), and the notes are rebuilt from the
|
||||
cache on the next run that finds Joplin up.
|
||||
|
||||
**One provider failing does not skip the rest.** Both installers loop over the
|
||||
providers in a single action rather than one action each, because systemd
|
||||
`oneshot` stops at the first failing `ExecStart` and Task Scheduler reports only
|
||||
the last action's result. Every provider is attempted; the run still exits
|
||||
non-zero if any failed. The loop is `scheduling/run-sync.sh` on Linux (the
|
||||
unit's `ExecStart`) and `scheduling\run-sync.ps1` on Windows (the task's action).
|
||||
|
||||
### Getting notified
|
||||
|
||||
A scheduled run is silent by default. Output goes to three pull-only places: the
|
||||
exporter's own log (`cache/logs/exporter.log`), the systemd journal on Linux
|
||||
(`journalctl --user -u aichat-sync.service`), and on Windows Task Scheduler's
|
||||
`LastTaskResult` plus `cache\logs\scheduled-sync.log`.
|
||||
|
||||
To have runs report back, set an [ntfy](https://ntfy.sh) topic in `.env`:
|
||||
|
||||
```bash
|
||||
NTFY_TOPIC=my-archive-topic
|
||||
```
|
||||
|
||||
Then check it works before waiting on a scheduled run:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter notify # show settings
|
||||
ai-chat-exporter notify --test # send a test push
|
||||
```
|
||||
|
||||
`sync` pushes a result whenever a topic is configured — a success carries the
|
||||
per-provider counts at low/default priority, a failure carries the reason at
|
||||
high priority with an alert tag, so a failed archive is distinguishable from a
|
||||
quiet one on your phone. `NTFY_NOTIFY=failure` notifies only on failure; `off`
|
||||
disables it; `--notify` / `--no-notify` override per run.
|
||||
|
||||
`sync` can only push from the end of a run it finished. A crash, an exit before
|
||||
the sync starts (the ToS gate, a cache error) or a launcher that can't build its
|
||||
venv sends nothing — and since each provider pushes separately, the ones that
|
||||
succeeded still say "OK", so a dead provider looks like a quiet day. The
|
||||
scheduler wrappers (`scheduling/run-sync.sh`, `scheduling\run-sync.ps1`) close
|
||||
that gap: any run that exits non-zero without the
|
||||
app having reported it gets a high-priority **FAILED** push naming the provider
|
||||
and, for a crash, the exception's class (`claude-code: crashed (RecursionError)`)
|
||||
— the class only, never its message, which can carry a conversation title. The
|
||||
traceback is in the journal on Linux and in `cache\logs\scheduled-sync.log` on
|
||||
Windows.
|
||||
|
||||
The message includes the **machine name**, which matters because both machines
|
||||
archive into one topic. It contains counts only — never conversation titles. A
|
||||
topic on public ntfy.sh is readable by anyone who knows its name, so if you want
|
||||
it private, self-host (`NTFY_SERVER`) or use an access-controlled topic with
|
||||
`NTFY_TOKEN`.
|
||||
|
||||
A notification is never fatal: if ntfy is unreachable, the run logs a warning and
|
||||
still reports its real exit code.
|
||||
|
||||
**Acknowledge the ToS notice once, interactively.** It's stored in the cache
|
||||
manifest per machine. Until then a scheduled run exits 1 with an explanation
|
||||
rather than hanging on a prompt no one can answer.
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
# Troubleshooting
|
||||
|
||||
What the cache does, what each failure means, and how to file a bug report.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [CLI Reference](cli.md) · [Scheduling](scheduling.md)
|
||||
|
||||
---
|
||||
|
||||
## How the Cache Works
|
||||
|
||||
The cache manifest lives at `cache/manifest.json` (inside the install directory) and records every exported conversation: its title, project, `updated_at` timestamp, output file path, and (after Joplin sync) the Joplin note ID.
|
||||
|
||||
On every `export` run:
|
||||
1. Fetch the full conversation list from the provider
|
||||
2. Compare each conversation's `updated_at` against the manifest
|
||||
3. Export only conversations that are new or have been updated
|
||||
4. Write each successfully exported conversation to the manifest **immediately** (not batched)
|
||||
|
||||
On every `joplin` run:
|
||||
1. Read the manifest to find conversations not yet synced to Joplin, or re-exported since last sync
|
||||
2. Push each pending Markdown file to Joplin (create or update)
|
||||
3. Store the Joplin note ID in the manifest so subsequent runs update rather than duplicate
|
||||
|
||||
**This design makes every run inherently resumable.** If the tool is interrupted for any reason — rate limit, network drop, Ctrl+C, crash — simply re-run the same command. It will skip already-processed conversations and continue from where it stopped.
|
||||
|
||||
A conversation whose assets failed transiently also carries `pending_media` (the
|
||||
asset refs still missing) and `pending_media_attempts`. While `pending_media` is
|
||||
set and the attempt cap is not reached, step 2 treats the conversation as stale
|
||||
so the next run re-fetches those assets — see
|
||||
[Embedded images and media](#embedded-images-and-media). A run that recovers them
|
||||
clears the key.
|
||||
|
||||
To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
|
||||
|
||||
To re-export a single conversation, delete its entry from `cache/manifest.json`
|
||||
and re-run. (There is deliberately no `cache --reset --conversation` flag; see
|
||||
`FUTURE.md`.)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `Authentication failed` (401, or 403 from Claude)
|
||||
Your session token has expired.
|
||||
- Run `ai-chat-exporter auth` to get a new token interactively
|
||||
- Or manually copy a fresh cookie value into your `.env` file
|
||||
|
||||
Note: neither token's expiry can be read client-side. Claude's `sessionKey` is an opaque string, and claude.ai reports an invalid one as **403** "Invalid authorization" (`account_session_invalid`), not 401. ChatGPT's token is an encrypted JWE; `doctor` reads the `error` field of `/api/auth/session` instead. See [When Tokens Expire](providers.md#when-tokens-expire).
|
||||
|
||||
### `429 Rate Limited`
|
||||
The tool automatically pauses, saves progress, and exits with a clear message showing how many conversations were exported vs remaining. Just re-run the same export command to resume — the cache picks up exactly where it left off.
|
||||
|
||||
### `HTTP 500` / `503` on an asset download
|
||||
The provider's asset storage, not your setup. Retries and the cross-run recovery
|
||||
described in [Embedded images and media](#embedded-images-and-media) handle this;
|
||||
the run reports `media failed` and the next run re-fetches. Measured 2026-10-20:
|
||||
two assets lost this way to a 500 and a 503 both downloaded cleanly an hour later.
|
||||
|
||||
### Joplin: "JOPLIN_API_TOKEN is not set"
|
||||
You need to configure the token before running the `joplin` command:
|
||||
1. Open Joplin desktop
|
||||
2. Go to Tools → Options → Web Clipper
|
||||
3. Enable the Web Clipper service
|
||||
4. Copy the Authorization token shown on that page
|
||||
5. Add `JOPLIN_API_TOKEN=<token>` to your `.env` file
|
||||
|
||||
### Joplin: "Joplin is not responding"
|
||||
Joplin desktop must be running when you run the `joplin` command. The Web Clipper service shuts down when Joplin is closed.
|
||||
|
||||
### Joplin: "Joplin rejected the API token (HTTP 401)"
|
||||
The token in `JOPLIN_API_TOKEN` doesn't match what Joplin expects. Get a fresh token from Joplin → Tools → Options → Web Clipper → Authorization token.
|
||||
|
||||
### Joplin: note timed out
|
||||
If you see a timeout error, Joplin took longer than `JOPLIN_REQUEST_TIMEOUT` seconds (default: 30) to respond. Possible causes:
|
||||
- The conversation is very large and Joplin is slow to index it
|
||||
- Joplin is busy syncing or loading a large library
|
||||
- Joplin has frozen — try restarting it
|
||||
|
||||
To increase the timeout: add `JOPLIN_REQUEST_TIMEOUT=60` to your `.env`.
|
||||
|
||||
### ChatGPT project conversations not appearing
|
||||
Make sure you've added the project IDs to `CHATGPT_PROJECT_IDS` in your `.env`. See [ChatGPT Projects](providers.md#chatgpt-projects) for how to find them. Project conversations are not included in the default conversation listing — they must be fetched separately.
|
||||
|
||||
### Schema warnings in logs (`Unexpected API response shape`)
|
||||
The provider's internal API may have changed. Run with `--debug`, sanitize the output (remove any personal content), and check the project's GitHub Issues for known fixes.
|
||||
|
||||
### Non-text content warnings
|
||||
Since v0.4.0, rich content is preserved as typed blocks in the export. ChatGPT voice transcripts render as text and audio assets as `📎 File attached` placeholders with size and duration metadata. Anything the extractor doesn't recognise renders as a visible `> ⚠️ Unsupported content` block naming the type and observed keys, *and* increments a counter in the post-export summary so you can tell whether real content is being silently skipped.
|
||||
|
||||
### Embedded images and media
|
||||
Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as ``. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
|
||||
|
||||
A failed asset **never fails the export** — the conversation is archived with its
|
||||
text placeholder, and the failure is tallied as `media failed` in the run summary
|
||||
and in the push notification. What happens next depends on *why* it failed:
|
||||
|
||||
| Bucket | Meaning | Retried? |
|
||||
|---|---|---|
|
||||
| `expired-or-missing` | 404 — the upload was deleted server-side | No. The bytes are gone. |
|
||||
| `forbidden` | 403 — metadata survives, bytes do not (see the 2026-08-17 investigation in `CHANGELOG.md`) | No. |
|
||||
| `download-error` | 5xx, a timeout, a network blip | **Yes** — within the run, then on up to 3 later runs |
|
||||
| `skipped-provider-unwell` | the run's circuit breaker was open | **Yes**, on a later run |
|
||||
| `unparseable-ref` / `write-error` | local problem, not the provider's | No |
|
||||
|
||||
**Within a run**, an asset download is two requests — one for a signed URL, one
|
||||
for the bytes — and each is retried on a 5xx: immediately once, then after about
|
||||
2, 4, 8 and 16 seconds with jitter, honouring `Retry-After` when the server sends
|
||||
one. Retries for a single asset are capped by a wall-clock budget so a hanging
|
||||
endpoint cannot consume the run's whole execution window.
|
||||
|
||||
**Across runs**, an asset that failed transiently is recorded against its
|
||||
conversation in the cache manifest, and the next run re-exports that conversation
|
||||
to re-fetch it. Only the missing asset is fetched — media already on disk is
|
||||
reused, and Joplin resource IDs are keyed by path, so the re-sync uploads just
|
||||
the new file. After 3 unsuccessful runs the asset is treated as lost and the
|
||||
conversation stops being re-exported.
|
||||
|
||||
**Circuit breaker.** If 5 transient failures land back to back, asset downloads
|
||||
stop for the rest of the run and the remaining assets are recorded for a later
|
||||
attempt. A provider's storage having a bad day should cost one quiet morning, not
|
||||
a run that overruns its two-hour scheduled window. Permanent failures never count
|
||||
toward this — a deleted upload says nothing about the provider's health.
|
||||
|
||||
### Tool output / hidden context collapsed
|
||||
Since v0.6.0, content that was invisible in the ChatGPT web UI is collapsed by default to one-line placeholders like `> 🔧 Tool output — file_search (15.0 KB) — omitted`. This is ChatGPT's file-retrieval tool re-injecting your attached files on every run — it can be 90% of a conversation's bytes while containing none of the dialogue. Custom Instructions collapse the same way (`> ℹ️ Hidden context`). The post-export summary lists every collapsed message by origin with the total KB omitted. To keep everything, set `EXPORTER_HIDDEN_CONTENT=full` in `.env` or pass `--hidden-content full`.
|
||||
|
||||
### Empty export / all conversations skipped
|
||||
No new or updated conversations since your last run. To verify: `ai-chat-exporter cache --show`. To force a full re-export: `ai-chat-exporter cache --clear`.
|
||||
|
||||
### Filing a bug report
|
||||
1. Run with `--debug`: `ai-chat-exporter export --debug 2>&1 | tee debug.log`
|
||||
2. Remove any personal conversation content from `debug.log`
|
||||
3. Open a GitHub Issue with the sanitized log and the exact command you ran
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "ai-chat-exporter"
|
||||
version = "0.9.0"
|
||||
version = "1.0.0"
|
||||
description = "Archive ChatGPT, Claude, Claude Code and Codex conversation history to Markdown for personal backup in Joplin"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
|
||||
@@ -35,6 +35,7 @@ $ErrorActionPreference = 'Stop'
|
||||
|
||||
$repo = Split-Path -Parent $PSScriptRoot
|
||||
$launcher = Join-Path $repo 'ai-chat-exporter.cmd'
|
||||
$wrapper = Join-Path $PSScriptRoot 'run-sync.ps1'
|
||||
|
||||
if ($Unregister) {
|
||||
Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false -ErrorAction SilentlyContinue
|
||||
@@ -42,22 +43,22 @@ if ($Unregister) {
|
||||
return
|
||||
}
|
||||
|
||||
if (-not (Test-Path $launcher)) {
|
||||
throw "Launcher not found at $launcher"
|
||||
foreach ($f in $launcher, $wrapper) {
|
||||
if (-not (Test-Path $f)) { throw "Not found: $f" }
|
||||
}
|
||||
|
||||
# A single action looping over the providers, rather than one action each.
|
||||
# Task Scheduler runs multiple actions in order but reports only the last one's
|
||||
# result, so a failure in an earlier provider would be invisible. The loop keeps
|
||||
# going after a failure and propagates a non-zero exit code.
|
||||
# One action running run-sync.ps1, rather than one action per provider: Task
|
||||
# Scheduler runs multiple actions in order but reports only the last one's
|
||||
# result, so a failure in an earlier provider would be invisible. The script
|
||||
# keeps going after a failure, exits non-zero if any provider failed, and pushes
|
||||
# a FAILED notification for any run that died without sending its own.
|
||||
#
|
||||
# /v:on and !RC! are required, not stylistic: cmd expands every %VAR% on a
|
||||
# command line *before* running any of it, so "exit /b %RC%" would report the
|
||||
# value RC had before the loop ever ran - i.e. always success. Delayed expansion
|
||||
# reads it at the point of use.
|
||||
$loop = ($Provider | ForEach-Object { "`"$launcher`" sync --provider $_ --joplin-optional || set RC=1" }) -join ' & '
|
||||
$taskArgs = "/v:on /c set RC=0 & $loop & exit /b !RC!"
|
||||
$actions = New-ScheduledTaskAction -Execute 'cmd.exe' `
|
||||
# Providers go as separate arguments: under -File, "chatgpt,claude" would arrive
|
||||
# as one string. -WindowStyle Hidden because the output now goes to
|
||||
# cache\logs\scheduled-sync.log, so the console window would only ever be blank.
|
||||
$taskArgs = "-NoProfile -NonInteractive -ExecutionPolicy Bypass -WindowStyle Hidden " +
|
||||
"-File `"$wrapper`" $($Provider -join ' ')"
|
||||
$actions = New-ScheduledTaskAction -Execute 'powershell.exe' `
|
||||
-Argument $taskArgs `
|
||||
-WorkingDirectory $repo
|
||||
|
||||
@@ -82,12 +83,12 @@ Register-ScheduledTask -TaskName $TaskName `
|
||||
Write-Host "Registered '$TaskName' - daily at $Time for: $($Provider -join ', ')"
|
||||
Write-Host ''
|
||||
Write-Host 'Command the task will run:'
|
||||
Write-Host " cmd.exe $taskArgs"
|
||||
Write-Host " powershell.exe $taskArgs"
|
||||
Write-Host " (working directory: $repo)"
|
||||
Write-Host ''
|
||||
Write-Host 'Next steps:'
|
||||
Write-Host " * Run it once now: Start-ScheduledTask -TaskName $TaskName"
|
||||
Write-Host " * Check the result: Get-ScheduledTaskInfo -TaskName $TaskName"
|
||||
Write-Host " * Read the log: Get-Content '$repo\cache\logs\exporter.log' -Tail 50"
|
||||
Write-Host " * Read the run output: Get-Content '$repo\cache\logs\scheduled-sync.log' -Tail 50"
|
||||
Write-Host ' * The terms-of-service notice must have been acknowledged'
|
||||
Write-Host ' interactively once on this machine, or the task exits 1.'
|
||||
|
||||
@@ -44,10 +44,12 @@ fi
|
||||
|
||||
[ ${#PROVIDERS[@]} -eq 0 ] && PROVIDERS=("all")
|
||||
|
||||
if [ ! -x "$REPO/ai-chat-exporter" ]; then
|
||||
echo "error: $REPO/ai-chat-exporter is missing or not executable." >&2
|
||||
for f in "$REPO/ai-chat-exporter" "$REPO/scheduling/run-sync.sh"; do
|
||||
if [ ! -x "$f" ]; then
|
||||
echo "error: $f is missing or not executable." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
mkdir -p "$UNIT_DIR"
|
||||
|
||||
@@ -65,12 +67,10 @@ mkdir -p "$UNIT_DIR"
|
||||
echo "Type=oneshot"
|
||||
echo "WorkingDirectory=$REPO"
|
||||
echo "Environment=AI_CHAT_EXPORTER_QUIET_CWD=1"
|
||||
# One ExecStart per provider would stop at the first failure, silently
|
||||
# skipping the rest — an expired ChatGPT token would mean codex never runs.
|
||||
# Loop instead, so every provider is attempted and the unit still reports
|
||||
# failure if any of them failed.
|
||||
printf 'ExecStart=/bin/sh -c '\''rc=0; for p in %s; do "$0" sync --provider "$p" --joplin-optional || rc=1; done; exit $rc'\'' %s\n' \
|
||||
"${PROVIDERS[*]}" "$REPO/ai-chat-exporter"
|
||||
echo "Environment=AICHAT_SYNC_UNIT=$NAME"
|
||||
# run-sync.sh attempts every provider even after one fails, and pushes a
|
||||
# FAILED notification for any run that died without sending its own.
|
||||
echo "ExecStart=$REPO/scheduling/run-sync.sh ${PROVIDERS[*]}"
|
||||
} > "$UNIT_DIR/$NAME.service"
|
||||
|
||||
# Persistent=true runs a missed schedule at the next boot — the machine being
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Run `ai-chat-exporter sync` once per provider - the action of the scheduled
|
||||
task that Register-AiChatSyncTask.ps1 creates.
|
||||
|
||||
.DESCRIPTION
|
||||
The Windows counterpart to run-sync.sh, and there for the same reason.
|
||||
|
||||
Every provider is attempted even after one fails, and the exit code is
|
||||
non-zero if any failed - Task Scheduler reports only the last action's
|
||||
result, so one action per provider would hide an earlier failure.
|
||||
|
||||
The app pushes its own ntfy result, but only from the end of a run it
|
||||
finished. A crash, a non-zero exit before the sync starts (the terms-of-
|
||||
service gate, a cache error) or a launcher that can't build its venv sends
|
||||
nothing, and because each provider pushes separately, the providers that did
|
||||
succeed still send "OK" - so a broken one looks like a quiet day. This
|
||||
script pushes a FAILED notification for any run that exited non-zero without
|
||||
the app having reported it. (Its "Sync completed with failures" banner
|
||||
prints right after its push.)
|
||||
|
||||
The push carries the provider, the exit code and, for a crash, the
|
||||
exception's class name - never its message. Same counts-only rule as
|
||||
src/notify.py: on a public ntfy topic anyone who guesses the name can read
|
||||
it, and exception text can carry conversation titles.
|
||||
|
||||
Task Scheduler keeps no output, so each run's output is appended to
|
||||
cache\logs\scheduled-sync.log (rolled over to .1 past 1 MB). That is where
|
||||
the traceback is.
|
||||
|
||||
.EXAMPLE
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File .\scheduling\run-sync.ps1 chatgpt claude
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(ValueFromRemainingArguments = $true)]
|
||||
[string[]]$Provider
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
$repo = Split-Path -Parent $PSScriptRoot
|
||||
$launcher = Join-Path $repo 'ai-chat-exporter.cmd'
|
||||
$envFile = Join-Path $repo '.env'
|
||||
$logDir = Join-Path $repo 'cache\logs'
|
||||
$runLog = Join-Path $logDir 'scheduled-sync.log'
|
||||
|
||||
# -File passes "chatgpt,claude" as one string; accept it as well as a list.
|
||||
$Provider = @($Provider | ForEach-Object { $_ -split ',' } | Where-Object { $_ })
|
||||
if ($Provider.Count -eq 0) { $Provider = @('all') }
|
||||
foreach ($p in $Provider) {
|
||||
# Each name is spliced into a cmd.exe command line below.
|
||||
if ($p -notmatch '^[A-Za-z0-9_-]+$') { throw "Invalid provider name: '$p'" }
|
||||
}
|
||||
|
||||
# Under Task Scheduler stdout is a file, not a console, so Python would encode
|
||||
# it in the ANSI code page and die on the first character outside it.
|
||||
$env:PYTHONIOENCODING = 'utf-8'
|
||||
|
||||
# NTFY_* as the app resolves them: the environment wins, then .env.
|
||||
function Get-EnvValue([string]$Name) {
|
||||
$value = [Environment]::GetEnvironmentVariable($Name)
|
||||
if (-not $value -and (Test-Path -LiteralPath $envFile)) {
|
||||
foreach ($line in Get-Content -LiteralPath $envFile) {
|
||||
if ($line -match "^\s*$Name\s*=\s*(.*)$") { $value = $Matches[1] }
|
||||
}
|
||||
if ($value) { $value = ($value -replace '\s+#.*$', '').Trim().Trim('"', "'") }
|
||||
}
|
||||
return "$value"
|
||||
}
|
||||
|
||||
function Send-FailurePush([string]$Body) {
|
||||
$topic = Get-EnvValue 'NTFY_TOPIC'
|
||||
$policy = (Get-EnvValue 'NTFY_NOTIFY').ToLower()
|
||||
if (-not $topic -or $policy -eq 'off') { return }
|
||||
$server = Get-EnvValue 'NTFY_SERVER'
|
||||
if (-not $server) { $server = 'https://ntfy.sh' }
|
||||
$token = Get-EnvValue 'NTFY_TOKEN'
|
||||
|
||||
# The same short name src/notify.py puts in its titles.
|
||||
$machine = [System.Net.Dns]::GetHostName().Split('.')[0]
|
||||
$headers = @{
|
||||
Title = "AI archive FAILED - $machine"
|
||||
Tags = 'rotating_light'
|
||||
Priority = 'high'
|
||||
}
|
||||
if ($token) { $headers.Authorization = "Bearer $token" }
|
||||
|
||||
try {
|
||||
# Windows PowerShell 5.1 on an older .NET may not offer TLS 1.2 by default.
|
||||
[Net.ServicePointManager]::SecurityProtocol = `
|
||||
[Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
|
||||
Invoke-WebRequest -UseBasicParsing -Method Post -TimeoutSec 15 `
|
||||
-Uri "$($server.TrimEnd('/'))/$topic" -Headers $headers `
|
||||
-ContentType 'text/plain; charset=utf-8' `
|
||||
-Body ([Text.Encoding]::UTF8.GetBytes($Body)) | Out-Null
|
||||
} catch {
|
||||
Write-Warning "run-sync: could not send the failure notification ($($_.Exception.Message))"
|
||||
}
|
||||
}
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $logDir | Out-Null
|
||||
if ((Test-Path -LiteralPath $runLog) -and (Get-Item -LiteralPath $runLog).Length -gt 1MB) {
|
||||
Move-Item -LiteralPath $runLog -Destination "$runLog.1" -Force
|
||||
}
|
||||
|
||||
$rc = 0
|
||||
foreach ($p in $Provider) {
|
||||
$out = [IO.Path]::GetTempFileName()
|
||||
try {
|
||||
# cmd.exe does the redirect, so stdout and stderr stay interleaved and
|
||||
# PowerShell's native-command stderr handling never sees either. /s strips
|
||||
# exactly the outer pair of quotes, leaving the quoted paths inside intact.
|
||||
$psi = New-Object System.Diagnostics.ProcessStartInfo
|
||||
$psi.FileName = if ($env:ComSpec) { $env:ComSpec } else { 'cmd.exe' }
|
||||
$psi.Arguments = "/d /s /c `"`"$launcher`" sync --provider $p --joplin-optional > `"$out`" 2>&1`""
|
||||
$psi.WorkingDirectory = $repo
|
||||
$psi.UseShellExecute = $false
|
||||
$proc = [Diagnostics.Process]::Start($psi)
|
||||
$proc.WaitForExit()
|
||||
$status = $proc.ExitCode
|
||||
|
||||
$text = [IO.File]::ReadAllText($out, [Text.Encoding]::UTF8)
|
||||
Write-Host $text
|
||||
Add-Content -LiteralPath $runLog -Encoding UTF8 -Value (
|
||||
"===== $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') sync --provider $p (exit $status) =====`r`n$text")
|
||||
|
||||
if ($status -ne 0) {
|
||||
$rc = 1
|
||||
if (-not $text.Contains('Sync completed with failures')) {
|
||||
$crash = [regex]::Matches($text, '(?m)^[A-Za-z_][A-Za-z0-9_.]*(Error|Exception)\b') |
|
||||
Select-Object -Last 1
|
||||
if ($crash) {
|
||||
$reason = "crashed ($($crash.Value))"
|
||||
} else {
|
||||
$reason = "exited $status before reporting a result"
|
||||
}
|
||||
Send-FailurePush "${p}: $reason`ncache\logs\scheduled-sync.log"
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
# This script failing is a run that reports nothing, too.
|
||||
$rc = 1
|
||||
Write-Warning "run-sync: $p - $($_.Exception.Message)"
|
||||
Send-FailurePush "${p}: run-sync.ps1 failed ($($_.Exception.GetType().Name))"
|
||||
} finally {
|
||||
Remove-Item -LiteralPath $out -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
exit $rc
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run `ai-chat-exporter sync` once per provider — the ExecStart of the systemd
|
||||
# unit that install-systemd-timer.sh writes.
|
||||
#
|
||||
# ./scheduling/run-sync.sh claude-code codex
|
||||
#
|
||||
# Every provider is attempted even after one fails, and the exit code is
|
||||
# non-zero if any failed — one ExecStart per provider would stop at the first.
|
||||
#
|
||||
# The app pushes its own ntfy result, but only from the end of a run it
|
||||
# finished. A crash, a non-zero exit before the sync starts (the terms-of-service
|
||||
# gate, a cache error) or a launcher that can't build its venv sends nothing, and
|
||||
# because each provider pushes separately, the providers that did succeed still
|
||||
# send "OK" — so a broken one looks like a quiet day. This script pushes a FAILED
|
||||
# notification for any run that exited non-zero without the app having reported
|
||||
# it. (Its "Sync completed with failures" banner prints right after its push.)
|
||||
#
|
||||
# The push carries the provider, the exit code and, for a crash, the exception's
|
||||
# class name — never its message. Same counts-only rule as src/notify.py: on a
|
||||
# public ntfy topic anyone who guesses the name can read it, and exception text
|
||||
# can carry conversation titles. The full traceback is in the journal.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
REPO="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
LAUNCHER="$REPO/ai-chat-exporter"
|
||||
|
||||
# NTFY_* as the app resolves them: the environment wins, then .env.
|
||||
env_value() {
|
||||
local name=$1 value=${!1:-}
|
||||
if [ -z "$value" ] && [ -f "$REPO/.env" ]; then
|
||||
value=$(sed -n "s/^[[:space:]]*$name[[:space:]]*=[[:space:]]*//p" "$REPO/.env" | tail -n 1)
|
||||
value=${value%%[[:space:]]#*}
|
||||
value=${value%"${value##*[![:space:]]}"}
|
||||
value=${value#[\"\']}
|
||||
value=${value%[\"\']}
|
||||
fi
|
||||
printf '%s' "$value"
|
||||
}
|
||||
|
||||
push_failure() {
|
||||
local body=$1 topic server token policy
|
||||
topic=$(env_value NTFY_TOPIC)
|
||||
policy=$(env_value NTFY_NOTIFY | tr '[:upper:]' '[:lower:]')
|
||||
if [ -z "$topic" ] || [ "$policy" = "off" ]; then
|
||||
return 0
|
||||
fi
|
||||
server=$(env_value NTFY_SERVER)
|
||||
server=${server:-https://ntfy.sh}
|
||||
token=$(env_value NTFY_TOKEN)
|
||||
|
||||
local args=(-fsS --max-time 15 -o /dev/null
|
||||
-H "Title: AI archive FAILED - $(hostname -s)"
|
||||
-H "Tags: rotating_light" -H "Priority: high"
|
||||
--data-binary "$body")
|
||||
[ -n "$token" ] && args+=(-H "Authorization: Bearer $token")
|
||||
curl "${args[@]}" "${server%/}/$topic" \
|
||||
|| echo "run-sync: could not send the failure notification" >&2
|
||||
}
|
||||
|
||||
[ $# -eq 0 ] && set -- all
|
||||
|
||||
rc=0
|
||||
for provider in "$@"; do
|
||||
out=$(mktemp)
|
||||
"$LAUNCHER" sync --provider "$provider" --joplin-optional 2>&1 | tee "$out"
|
||||
status=${PIPESTATUS[0]}
|
||||
if [ "$status" -ne 0 ]; then
|
||||
rc=1
|
||||
if ! grep -q "Sync completed with failures" "$out"; then
|
||||
crash=$(grep -oE '^[A-Za-z_][A-Za-z0-9_.]*(Error|Exception)\b' "$out" | tail -n 1)
|
||||
if [ -n "$crash" ]; then
|
||||
reason="crashed ($crash)"
|
||||
else
|
||||
reason="exited $status before reporting a result"
|
||||
fi
|
||||
push_failure "$provider: $reason
|
||||
journalctl --user -u ${AICHAT_SYNC_UNIT:-aichat-sync} -n 100"
|
||||
fi
|
||||
fi
|
||||
rm -f "$out"
|
||||
done
|
||||
exit "$rc"
|
||||
@@ -10,6 +10,12 @@ from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# How many runs may re-export a conversation to re-fetch a transiently failed
|
||||
# asset before giving up on it. Three mornings is enough to ride out a storage
|
||||
# incident; past that the asset is treated as lost and the conversation stops
|
||||
# being re-exported (the record of which assets were lost is kept).
|
||||
MAX_MEDIA_RETRY_RUNS = 3
|
||||
|
||||
MANIFEST_VERSION = 1
|
||||
|
||||
|
||||
@@ -57,6 +63,33 @@ class Cache:
|
||||
logger.debug("[cache] MISS %s/%s", provider, conv_id[:8])
|
||||
return False
|
||||
|
||||
# Assets the provider fumbled transiently (a 5xx or timeout on the
|
||||
# signed URL) are worth asking for again: the conversation itself is
|
||||
# up to date, but it is missing bytes that almost certainly still
|
||||
# exist. Re-exporting is cheap — media already on disk is reused and
|
||||
# Joplin resources are keyed by path — so only the missing asset is
|
||||
# fetched. Capped, because an asset that fails this way every single
|
||||
# day would otherwise re-export and re-sync its conversation forever.
|
||||
pending = entry.get("pending_media")
|
||||
if pending:
|
||||
attempts = int(entry.get("pending_media_attempts") or 0)
|
||||
if attempts <= MAX_MEDIA_RETRY_RUNS:
|
||||
logger.debug(
|
||||
"[cache] RETRY-MEDIA %s/%s (%d asset(s), retry %d/%d)",
|
||||
provider,
|
||||
conv_id[:8],
|
||||
len(pending),
|
||||
attempts,
|
||||
MAX_MEDIA_RETRY_RUNS,
|
||||
)
|
||||
return False
|
||||
logger.debug(
|
||||
"[cache] HIT (media retries exhausted) %s/%s — %d asset(s) lost",
|
||||
provider,
|
||||
conv_id[:8],
|
||||
len(pending),
|
||||
)
|
||||
|
||||
cached_updated = entry.get("updated_at", "")
|
||||
if not cached_updated or not updated_at:
|
||||
logger.debug("[cache] HIT (no date comparison) %s/%s", provider, conv_id[:8])
|
||||
@@ -102,6 +135,16 @@ class Cache:
|
||||
if carried in prev:
|
||||
entry[carried] = prev[carried]
|
||||
|
||||
# Assets that failed transiently this run. Deliberately NOT carried
|
||||
# forward from `prev`: a run that resolved them must clear the flag,
|
||||
# which is what makes "pending" mean "still missing" rather than
|
||||
# "was missing once". The attempt counter does carry, so repeated
|
||||
# failure eventually stops costing a daily re-export.
|
||||
pending = [str(r) for r in (metadata.get("pending_media") or [])]
|
||||
if pending:
|
||||
entry["pending_media"] = pending
|
||||
entry["pending_media_attempts"] = int(prev.get("pending_media_attempts") or 0) + 1
|
||||
|
||||
self._data[provider][conv_id] = entry
|
||||
self._data["last_run"] = datetime.now(tz=timezone.utc).isoformat()
|
||||
self._save()
|
||||
|
||||
+18
-4
@@ -258,7 +258,10 @@ def _auth_claude(os_name: str) -> None:
|
||||
console.print("2. Press [bold]F12[/bold] to open DevTools → Application tab.")
|
||||
console.print("3. Expand [bold]Cookies[/bold] → [bold]https://claude.ai[/bold]")
|
||||
console.print("4. Find [bold]sessionKey[/bold] → copy the Value.")
|
||||
console.print(" (Note: Claude tokens expire after ~30 days; a 401 error is the only signal.)")
|
||||
console.print(
|
||||
" (Note: Claude tokens expire after ~30 days; an API rejection — 403 "
|
||||
"'Invalid authorization' — is the only signal.)"
|
||||
)
|
||||
console.print("5. Paste it below (input is hidden).\n")
|
||||
|
||||
key = click.prompt("Claude session key", hide_input=True, default="", show_default=False).strip()
|
||||
@@ -869,8 +872,13 @@ def export(
|
||||
# Single LossReport tracks data-loss visibility across all providers in this run.
|
||||
loss_report = LossReport()
|
||||
|
||||
# One breaker for the whole run: a provider's asset storage being unwell is
|
||||
# a run-level fact, not a per-conversation one.
|
||||
from src.media import MediaCircuit
|
||||
media_circuit = MediaCircuit()
|
||||
|
||||
for prov_name, prov_instance in providers_to_run:
|
||||
summary[prov_name] = {"exported": 0, "skipped": 0, "failed": 0}
|
||||
summary[prov_name] = {"exported": 0, "skipped": 0, "failed": 0, "media_failed": 0}
|
||||
|
||||
console.print(f"\n[bold cyan][{prov_name.upper()}][/bold cyan] Fetching conversation list…")
|
||||
|
||||
@@ -955,16 +963,19 @@ def export(
|
||||
|
||||
# Download assets (images by default) so the renderers
|
||||
# can inline them. Failures keep placeholders; never fatal.
|
||||
from src.media import MediaOutcome, resolve_media
|
||||
media_outcome = MediaOutcome()
|
||||
if cfg.download_media != "off":
|
||||
from src.media import resolve_media
|
||||
resolve_media(
|
||||
media_outcome = resolve_media(
|
||||
normalized,
|
||||
prov_instance,
|
||||
export_base,
|
||||
structure,
|
||||
cfg.download_media,
|
||||
loss_report,
|
||||
circuit=media_circuit,
|
||||
)
|
||||
summary[prov_name]["media_failed"] += media_outcome.failed
|
||||
|
||||
exported_path: Path | None = None
|
||||
if md_exporter:
|
||||
@@ -979,6 +990,9 @@ def export(
|
||||
"created_at": normalized.get("created_at", ""),
|
||||
"updated_at": normalized.get("updated_at", ""),
|
||||
"file_path": str(exported_path) if exported_path else "",
|
||||
# Empty clears the flag, so a run that recovered the
|
||||
# asset stops the conversation being re-exported.
|
||||
"pending_media": media_outcome.retryable_refs,
|
||||
})
|
||||
summary[prov_name]["exported"] += 1
|
||||
progress.advance(task)
|
||||
|
||||
+131
-13
@@ -12,13 +12,28 @@ Policy (``EXPORTER_DOWNLOAD_MEDIA``):
|
||||
are already in the exports as text)
|
||||
off — leave placeholders untouched
|
||||
|
||||
Failures (expired assets, network) keep the existing placeholder and are
|
||||
counted in the run summary — never fatal to the export.
|
||||
Failures keep the existing placeholder and are counted in the run summary —
|
||||
never fatal to the export. They split two ways, and the split is what decides
|
||||
whether the asset is ever seen again:
|
||||
|
||||
permanent — ``expired-or-missing`` (404) and ``forbidden`` (403). The bytes
|
||||
are gone server-side; retrying achieves nothing. Investigated
|
||||
exhaustively 2026-08-17, see ``_classify_failure``.
|
||||
transient — everything else: a 5xx from the signed URL, a timeout, a
|
||||
network blip. The asset almost certainly still exists, so its
|
||||
ref is handed back for the cache to remember and the next
|
||||
scheduled run re-attempts it.
|
||||
|
||||
A run also carries a ``MediaCircuit``: once enough *transient* failures land
|
||||
back to back, asset downloads stop for the rest of the run. A storage incident
|
||||
fails every asset in the archive, and a patient retry ladder times that many
|
||||
failures over is how a scheduled run exceeds its execution window.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
from src.blocks import BLOCK_TYPE_FILE_PLACEHOLDER, BLOCK_TYPE_IMAGE_PLACEHOLDER
|
||||
@@ -33,6 +48,19 @@ MEDIA_ALL = "all"
|
||||
MEDIA_OFF = "off"
|
||||
VALID_MEDIA_POLICIES = {MEDIA_IMAGES, MEDIA_ALL, MEDIA_OFF}
|
||||
|
||||
# Failure buckets whose asset is worth asking for again on a later run. The
|
||||
# permanent buckets are deliberately excluded: this archive holds 19 assets
|
||||
# whose bytes OpenAI lost in a 2026-07-14 Library migration, and retrying
|
||||
# those every morning forever would be pure noise — and would trip the
|
||||
# circuit breaker below on a server that is perfectly healthy.
|
||||
TRANSIENT_FAILURE_BUCKETS = frozenset({"download-error"})
|
||||
|
||||
# Consecutive transient failures before asset downloads stop for this run.
|
||||
MEDIA_CIRCUIT_THRESHOLD = 5
|
||||
|
||||
# Bucket recorded for assets skipped because the breaker had already tripped.
|
||||
CIRCUIT_OPEN_BUCKET = "skipped-provider-unwell"
|
||||
|
||||
_EXT_BY_MIME = {
|
||||
"image/png": ".png",
|
||||
"image/jpeg": ".jpg",
|
||||
@@ -62,6 +90,64 @@ def resolve_media_policy() -> str:
|
||||
return value
|
||||
|
||||
|
||||
@dataclass
|
||||
class MediaOutcome:
|
||||
"""What one conversation's media pass achieved.
|
||||
|
||||
``retryable_refs`` is the part that matters beyond the summary line: the
|
||||
caller records it against the conversation in the cache manifest, which is
|
||||
what lets a later run re-attempt an asset the provider fumbled. Without
|
||||
it a transient 5xx lost the asset permanently, because the conversation was
|
||||
marked exported and never looked at again.
|
||||
"""
|
||||
|
||||
downloaded: int = 0
|
||||
failed: int = 0
|
||||
retryable_refs: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
class MediaCircuit:
|
||||
"""Run-scoped breaker that stops asset downloads when the provider is unwell.
|
||||
|
||||
One run-level object, shared by every conversation in the run. Counts
|
||||
*consecutive transient* failures: a permanent failure (a deleted upload)
|
||||
says nothing about the provider's health and must not trip it, and any
|
||||
success resets the count, so ordinary scattered losses never trip it either.
|
||||
|
||||
Once tripped it stays tripped for the run. The alternative — keep trying
|
||||
every asset through a storage incident — is what pushes a scheduled run
|
||||
past its execution window (two hours, in the Windows task definition).
|
||||
Assets skipped while open are still reported as retryable, so the next run
|
||||
picks them up rather than losing them.
|
||||
"""
|
||||
|
||||
def __init__(self, threshold: int = MEDIA_CIRCUIT_THRESHOLD) -> None:
|
||||
self.threshold = threshold
|
||||
self.consecutive = 0
|
||||
self.tripped = False
|
||||
self.skipped = 0
|
||||
|
||||
def record_success(self) -> None:
|
||||
self.consecutive = 0
|
||||
|
||||
def record_failure(self, bucket: str) -> None:
|
||||
"""Count a failure; only transient ones count toward tripping."""
|
||||
if bucket not in TRANSIENT_FAILURE_BUCKETS:
|
||||
return
|
||||
self.consecutive += 1
|
||||
if not self.tripped and self.consecutive >= self.threshold:
|
||||
self.tripped = True
|
||||
logger.error(
|
||||
"[media] %d consecutive transient asset failures — skipping "
|
||||
"further downloads this run. Affected assets are recorded and "
|
||||
"will be re-attempted on the next run.",
|
||||
self.consecutive,
|
||||
)
|
||||
|
||||
def record_skip(self) -> None:
|
||||
self.skipped += 1
|
||||
|
||||
|
||||
def resolve_media(
|
||||
normalized: dict,
|
||||
provider,
|
||||
@@ -69,19 +155,28 @@ def resolve_media(
|
||||
structure: str,
|
||||
policy: str,
|
||||
report: LossReport,
|
||||
) -> int:
|
||||
"""Download assets for one normalized conversation. Returns download count.
|
||||
circuit: "MediaCircuit | None" = None,
|
||||
) -> MediaOutcome:
|
||||
"""Download assets for one normalized conversation.
|
||||
|
||||
Mutates placeholder blocks in place (adds ``local_path``). Idempotent:
|
||||
an asset already on disk is annotated without an API call.
|
||||
an asset already on disk is annotated without an API call, which is also
|
||||
what makes a re-export cheap when only one asset needs re-fetching.
|
||||
|
||||
Returns:
|
||||
A ``MediaOutcome``. ``retryable_refs`` is non-empty when an asset
|
||||
failed transiently (or was skipped by an open circuit) and should be
|
||||
re-attempted on a later run.
|
||||
"""
|
||||
outcome = MediaOutcome()
|
||||
|
||||
if policy == MEDIA_OFF:
|
||||
return 0
|
||||
return outcome
|
||||
|
||||
download = getattr(provider, "download_asset", None)
|
||||
if download is None:
|
||||
# Provider has no remote assets (e.g. claude-code local transcripts).
|
||||
return 0
|
||||
return outcome
|
||||
|
||||
wanted_types = {BLOCK_TYPE_IMAGE_PLACEHOLDER}
|
||||
if policy == MEDIA_ALL:
|
||||
@@ -94,7 +189,7 @@ def resolve_media(
|
||||
if block.get("type") in wanted_types and block.get("ref")
|
||||
]
|
||||
if not targets:
|
||||
return 0
|
||||
return outcome
|
||||
|
||||
# Same path computation the exporter uses, so media/ lands beside the .md.
|
||||
filename = generate_filename(
|
||||
@@ -112,12 +207,12 @@ def resolve_media(
|
||||
).parent
|
||||
media_dir = conv_dir / "media"
|
||||
|
||||
downloaded = 0
|
||||
for block in targets:
|
||||
ref = block["ref"]
|
||||
file_id = _safe_asset_name(provider, ref)
|
||||
if not file_id:
|
||||
report.record_media_failed("unparseable-ref")
|
||||
outcome.failed += 1
|
||||
continue
|
||||
|
||||
existing = _find_existing(media_dir, file_id)
|
||||
@@ -125,13 +220,32 @@ def resolve_media(
|
||||
block["local_path"] = f"media/{existing.name}"
|
||||
continue
|
||||
|
||||
# Breaker open: do not touch the network, but keep the ref so the
|
||||
# asset is re-attempted rather than silently abandoned.
|
||||
if circuit is not None and circuit.tripped:
|
||||
circuit.record_skip()
|
||||
report.record_media_failed(CIRCUIT_OPEN_BUCKET)
|
||||
outcome.failed += 1
|
||||
outcome.retryable_refs.append(ref)
|
||||
continue
|
||||
|
||||
try:
|
||||
content, mime, file_name = download(ref)
|
||||
except ProviderError as e:
|
||||
bucket = _classify_failure(e)
|
||||
logger.warning(
|
||||
"[media] Could not download %s: %s", ref[:60], e.original
|
||||
"[media] Could not download %s: %s (%s%s)",
|
||||
ref[:60],
|
||||
e.original,
|
||||
bucket,
|
||||
", will retry next run" if bucket in TRANSIENT_FAILURE_BUCKETS else "",
|
||||
)
|
||||
report.record_media_failed(_classify_failure(e))
|
||||
report.record_media_failed(bucket)
|
||||
outcome.failed += 1
|
||||
if bucket in TRANSIENT_FAILURE_BUCKETS:
|
||||
outcome.retryable_refs.append(ref)
|
||||
if circuit is not None:
|
||||
circuit.record_failure(bucket)
|
||||
continue
|
||||
|
||||
ext = _pick_extension(mime, file_name)
|
||||
@@ -141,13 +255,17 @@ def resolve_media(
|
||||
except OSError as e:
|
||||
logger.error("[media] Could not write %s: %s", target, e)
|
||||
report.record_media_failed("write-error")
|
||||
outcome.failed += 1
|
||||
continue
|
||||
|
||||
block["local_path"] = f"media/{target.name}"
|
||||
report.record_media_downloaded(len(content))
|
||||
downloaded += 1
|
||||
outcome.downloaded += 1
|
||||
if circuit is not None:
|
||||
circuit.record_success()
|
||||
|
||||
return outcome
|
||||
|
||||
return downloaded
|
||||
|
||||
|
||||
def _classify_failure(error: ProviderError) -> str:
|
||||
|
||||
@@ -148,10 +148,18 @@ def format_summary(
|
||||
exported = counts.get("exported", 0)
|
||||
skipped = counts.get("skipped", 0)
|
||||
failed = counts.get("failed", 0)
|
||||
media_failed = counts.get("media_failed", 0)
|
||||
total_exported += exported
|
||||
line = f"{prov}: {exported} exported, {skipped} up to date"
|
||||
if failed:
|
||||
line += f", {failed} FAILED"
|
||||
# Media failures do not fail the run — the transcript is archived and a
|
||||
# transient asset is re-attempted on a later run — but they were
|
||||
# invisible here until 2026-10-20, when two assets were lost in a run
|
||||
# that pushed a clean result. A count is enough to prompt a log check;
|
||||
# asset ids stay out, like conversation titles (see module docstring).
|
||||
if media_failed:
|
||||
line += f", {media_failed} media failed"
|
||||
lines.append(line)
|
||||
|
||||
synced = sum(
|
||||
|
||||
+159
-42
@@ -41,6 +41,19 @@ MAX_RETRIES = 3
|
||||
BACKOFF_BASE = 2.0
|
||||
BACKOFF_MAX = 60.0
|
||||
|
||||
# Asset (media) download retry policy. The asset endpoints flake independently
|
||||
# of the conversation API: measured 2026-10-20 in one scheduled run, a signed
|
||||
# URL returned 500 on one image and 503 on another, while the metadata hop for
|
||||
# the same assets succeeded. They therefore get a more patient ladder than the
|
||||
# conversation API — an immediate retry, then ~2/4/8/16s with jitter — bounded
|
||||
# by a wall-clock budget, because the other failure mode is a 40s timeout and
|
||||
# an unbounded ladder over those would eat a scheduled run's whole window.
|
||||
MEDIA_MAX_RETRIES = 5
|
||||
MEDIA_TIME_BUDGET = 60.0
|
||||
# Shorter read timeout than REQUEST_TIMEOUT: an asset that has not started
|
||||
# sending bytes in 20s is not about to, and the budget has retries to fund.
|
||||
MEDIA_REQUEST_TIMEOUT = (10, 20)
|
||||
|
||||
# EXPORTER_HIDDEN_CONTENT policy — what to do with content that was invisible
|
||||
# in the provider's UI (retrieval dumps, hidden-flagged context, agent tool
|
||||
# traffic). Shared by all providers.
|
||||
@@ -316,35 +329,73 @@ class BaseProvider(ABC):
|
||||
|
||||
return all_convs
|
||||
|
||||
def _make_request(
|
||||
def _send_with_retries(
|
||||
self,
|
||||
method: str,
|
||||
url: str,
|
||||
debug_log_body: bool = False,
|
||||
max_retries: int | None = None,
|
||||
immediate_first_retry: bool = False,
|
||||
time_budget: float | None = None,
|
||||
**kwargs: Any,
|
||||
) -> dict | list:
|
||||
"""Make an authenticated HTTP request with retry logic and unified error handling.
|
||||
) -> Any:
|
||||
"""Send one request with retry/backoff, returning the raw response.
|
||||
|
||||
The transport half of ``_make_request``, split out so callers that want
|
||||
**bytes** rather than JSON get identical retry behaviour. Before this
|
||||
existed, the signed-URL hop of an asset download was a bare
|
||||
``session.request`` with no 5xx retry at all, and its timeouts escaped
|
||||
as ``OSError`` (curl_cffi's ``RequestException`` subclasses it) to be
|
||||
mis-reported by the export loop as "File write failed".
|
||||
|
||||
Args:
|
||||
method: HTTP method ("GET", "POST", etc.)
|
||||
url: Full URL.
|
||||
debug_log_body: If True, log redacted response body at DEBUG level.
|
||||
max_retries: Retry ceiling; defaults to ``MAX_RETRIES``.
|
||||
immediate_first_retry: Retry a 5xx once with no wait before backing
|
||||
off. A single bad backend node is the most common shape of a
|
||||
transient 5xx, and a zero-cost retry catches it.
|
||||
time_budget: Seconds after which no further attempt starts. The last
|
||||
attempt's read timeout is clamped to what remains, so the real
|
||||
bound is the budget plus the connect timeout. Used to keep an
|
||||
endpoint that is timing out (40s a go, measured 2026-10-20) from
|
||||
consuming a scheduled run's whole execution window.
|
||||
**kwargs: Passed to requests.Session.request().
|
||||
|
||||
Returns:
|
||||
Parsed JSON response body.
|
||||
The response object, for the caller to read as JSON or bytes.
|
||||
|
||||
Raises:
|
||||
ProviderError: On 401, exhausted retries, or unrecoverable errors.
|
||||
ProviderError: On an authentication failure, exhausted retries, or
|
||||
unrecoverable errors.
|
||||
"""
|
||||
ceiling = MAX_RETRIES if max_retries is None else max_retries
|
||||
kwargs.setdefault("timeout", REQUEST_TIMEOUT)
|
||||
base_timeout = kwargs["timeout"]
|
||||
started = time.monotonic()
|
||||
|
||||
def _remaining() -> float | None:
|
||||
if time_budget is None:
|
||||
return None
|
||||
return time_budget - (time.monotonic() - started)
|
||||
|
||||
def _budget_exhausted(wait: float = 0.0) -> bool:
|
||||
left = _remaining()
|
||||
return left is not None and left - wait <= 0
|
||||
|
||||
attempt = 0
|
||||
last_exc: Exception | None = None
|
||||
|
||||
while attempt <= MAX_RETRIES:
|
||||
while attempt <= ceiling:
|
||||
attempt += 1
|
||||
self._pace()
|
||||
|
||||
# Clamp the read timeout to the remaining budget so the final
|
||||
# attempt cannot overrun it by a whole timeout period.
|
||||
left = _remaining()
|
||||
if left is not None and isinstance(base_timeout, tuple):
|
||||
connect_to, read_to = base_timeout
|
||||
kwargs["timeout"] = (connect_to, max(1.0, min(read_to, left)))
|
||||
|
||||
start = time.monotonic()
|
||||
|
||||
try:
|
||||
@@ -361,30 +412,35 @@ class BaseProvider(ABC):
|
||||
elapsed_ms,
|
||||
)
|
||||
|
||||
# ── 401: token expired / invalid ──────────────────────────
|
||||
if response.status_code == 401:
|
||||
self._handle_401()
|
||||
# _handle_401 raises ProviderError — this line never runs
|
||||
# ── Auth failure: token expired / invalid ──────────────────
|
||||
# Not keyed on 401 alone: a provider is free to answer an
|
||||
# invalid session with some other status, and claude.ai does
|
||||
# (403). Ask the provider rather than assuming.
|
||||
if self._is_auth_failure(response):
|
||||
self._handle_auth_failure(response)
|
||||
# _handle_auth_failure raises — this line never runs
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"{method} {url}",
|
||||
RuntimeError("401 Unauthorized"),
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} — not authenticated"
|
||||
),
|
||||
)
|
||||
|
||||
# ── 429: rate limited ──────────────────────────────────────
|
||||
if response.status_code == 429:
|
||||
if attempt > MAX_RETRIES:
|
||||
wait = self._backoff_wait(response, attempt)
|
||||
if attempt > ceiling or _budget_exhausted(wait):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"{method} {url}",
|
||||
RuntimeError("429 Rate Limited — max retries exhausted"),
|
||||
)
|
||||
wait = self._backoff_wait(response, attempt)
|
||||
logger.warning(
|
||||
"[%s] Rate limited. Retry %d/%d in %.1fs",
|
||||
self.provider_name,
|
||||
attempt,
|
||||
MAX_RETRIES,
|
||||
ceiling,
|
||||
wait,
|
||||
)
|
||||
time.sleep(wait)
|
||||
@@ -392,23 +448,26 @@ class BaseProvider(ABC):
|
||||
|
||||
# ── 5xx: server error ──────────────────────────────────────
|
||||
if response.status_code >= 500:
|
||||
if attempt > MAX_RETRIES:
|
||||
wait = self._backoff_wait(
|
||||
response, attempt, immediate_first=immediate_first_retry
|
||||
)
|
||||
if attempt > ceiling or _budget_exhausted(wait):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"{method} {url}",
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} after {MAX_RETRIES} retries"
|
||||
f"HTTP {response.status_code} after {attempt - 1} retries"
|
||||
),
|
||||
)
|
||||
wait = self._backoff_wait(None, attempt)
|
||||
logger.warning(
|
||||
"[%s] HTTP %d. Retry %d/%d in %.1fs",
|
||||
self.provider_name,
|
||||
response.status_code,
|
||||
attempt,
|
||||
MAX_RETRIES,
|
||||
ceiling,
|
||||
wait,
|
||||
)
|
||||
if wait:
|
||||
time.sleep(wait)
|
||||
continue
|
||||
|
||||
@@ -427,41 +486,32 @@ class BaseProvider(ABC):
|
||||
),
|
||||
)
|
||||
|
||||
# ── Success ────────────────────────────────────────────────
|
||||
body = response.json()
|
||||
|
||||
if debug_log_body:
|
||||
logger.debug(
|
||||
"[%s] Response body (redacted): %s",
|
||||
self.provider_name,
|
||||
redact_secrets(body),
|
||||
)
|
||||
|
||||
return body
|
||||
return response
|
||||
|
||||
except ProviderError:
|
||||
raise
|
||||
|
||||
except (requests.ConnectionError, requests.Timeout, _CurlConnectionError, _CurlTimeout) as e:
|
||||
last_exc = e
|
||||
if attempt > MAX_RETRIES:
|
||||
wait = self._backoff_wait(None, attempt, immediate_first=immediate_first_retry)
|
||||
if attempt > ceiling or _budget_exhausted(wait):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"{method} {url}",
|
||||
RuntimeError(
|
||||
f"Network error after {MAX_RETRIES} retries: {e}. "
|
||||
f"Network error after {attempt - 1} retries: {e}. "
|
||||
"Check your internet connection."
|
||||
),
|
||||
) from e
|
||||
wait = self._backoff_wait(None, attempt)
|
||||
logger.warning(
|
||||
"[%s] Network error (attempt %d/%d): %s. Retrying in %.1fs",
|
||||
self.provider_name,
|
||||
attempt,
|
||||
MAX_RETRIES,
|
||||
ceiling,
|
||||
e,
|
||||
wait,
|
||||
)
|
||||
if wait:
|
||||
time.sleep(wait)
|
||||
|
||||
except (requests.HTTPError, _CurlHTTPError) as e:
|
||||
@@ -476,11 +526,59 @@ class BaseProvider(ABC):
|
||||
last_exc or RuntimeError("Unknown error"),
|
||||
)
|
||||
|
||||
def _handle_401(self) -> None:
|
||||
"""Log a clear human-readable message for a 401 and raise ProviderError."""
|
||||
def _make_request(
|
||||
self,
|
||||
method: str,
|
||||
url: str,
|
||||
debug_log_body: bool = False,
|
||||
**kwargs: Any,
|
||||
) -> dict | list:
|
||||
"""Make an authenticated HTTP request and return its parsed JSON body.
|
||||
|
||||
Retry, backoff and error handling live in ``_send_with_retries``; this
|
||||
adds JSON parsing and optional body logging.
|
||||
|
||||
Args:
|
||||
method: HTTP method ("GET", "POST", etc.)
|
||||
url: Full URL.
|
||||
debug_log_body: If True, log redacted response body at DEBUG level.
|
||||
**kwargs: Passed through (see ``_send_with_retries``).
|
||||
|
||||
Returns:
|
||||
Parsed JSON response body.
|
||||
|
||||
Raises:
|
||||
ProviderError: On an authentication failure, exhausted retries, or
|
||||
unrecoverable errors.
|
||||
"""
|
||||
response = self._send_with_retries(method, url, **kwargs)
|
||||
body = response.json()
|
||||
|
||||
if debug_log_body:
|
||||
logger.debug(
|
||||
"[%s] Response body (redacted): %s",
|
||||
self.provider_name,
|
||||
redact_secrets(body),
|
||||
)
|
||||
|
||||
return body
|
||||
|
||||
def _is_auth_failure(self, response: Any) -> bool:
|
||||
"""Whether this response means the stored credential is not valid.
|
||||
|
||||
Defaults to 401. Providers whose API reports an invalid session with a
|
||||
different status override this — claude.ai returns 403, so keying auth
|
||||
handling on 401 alone reports an expired key as a generic permission
|
||||
error and never tells the user to refresh it.
|
||||
"""
|
||||
return bool(response.status_code == 401)
|
||||
|
||||
def _handle_auth_failure(self, response: Any) -> None:
|
||||
"""Log a clear human-readable message for an auth failure and raise."""
|
||||
# Subclasses override to include provider-specific cookie name
|
||||
msg = (
|
||||
f"[{self.provider_name}] Authentication failed (401 Unauthorized). "
|
||||
f"[{self.provider_name}] Authentication failed "
|
||||
f"(HTTP {response.status_code}). "
|
||||
"Your session token has likely expired. "
|
||||
"Run 'ai-chat-exporter auth' to refresh your token."
|
||||
)
|
||||
@@ -488,12 +586,28 @@ class BaseProvider(ABC):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
"authentication",
|
||||
RuntimeError("401 Unauthorized — token expired"),
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} — session token expired or invalid"
|
||||
),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _backoff_wait(response: requests.Response | None, attempt: int) -> float:
|
||||
"""Compute exponential backoff with jitter."""
|
||||
def _backoff_wait(
|
||||
response: requests.Response | None,
|
||||
attempt: int,
|
||||
immediate_first: bool = False,
|
||||
) -> float:
|
||||
"""Compute exponential backoff with jitter.
|
||||
|
||||
``Retry-After`` wins when the server sends one — including ahead of an
|
||||
immediate retry, because a server naming a time knows better than we do
|
||||
(503s commonly carry it).
|
||||
|
||||
``immediate_first`` makes the first retry free and shifts the ladder
|
||||
down one step, giving 0, 2, 4, 8, 16s rather than 2, 4, 8, 16, 32s.
|
||||
Jitter is never dropped: a storage incident fails many assets at once,
|
||||
and un-jittered retries would hit a recovering backend in lockstep.
|
||||
"""
|
||||
if response is not None:
|
||||
retry_after = response.headers.get("Retry-After")
|
||||
if retry_after:
|
||||
@@ -501,7 +615,10 @@ class BaseProvider(ABC):
|
||||
return float(retry_after)
|
||||
except ValueError:
|
||||
pass
|
||||
base = min(BACKOFF_BASE ** attempt, BACKOFF_MAX)
|
||||
if immediate_first and attempt <= 1:
|
||||
return 0.0
|
||||
exponent = attempt - 1 if immediate_first else attempt
|
||||
base = min(BACKOFF_BASE ** exponent, BACKOFF_MAX)
|
||||
jitter = random.uniform(0, base * 0.3)
|
||||
return base + jitter
|
||||
|
||||
|
||||
@@ -51,6 +51,9 @@ from src.providers.base import (
|
||||
HIDDEN_CONTENT_FULL,
|
||||
HIDDEN_CONTENT_OMIT,
|
||||
HIDDEN_CONTENT_PLACEHOLDER,
|
||||
MEDIA_MAX_RETRIES,
|
||||
MEDIA_REQUEST_TIMEOUT,
|
||||
MEDIA_TIME_BUDGET,
|
||||
ProviderError,
|
||||
REQUEST_TIMEOUT,
|
||||
VALID_HIDDEN_CONTENT_POLICIES,
|
||||
@@ -307,9 +310,9 @@ class ChatGPTProvider(BaseProvider):
|
||||
)
|
||||
return access_token
|
||||
|
||||
def _handle_401(self) -> None:
|
||||
def _handle_auth_failure(self, response: Any) -> None:
|
||||
msg = (
|
||||
"[chatgpt] Authentication failed (401 Unauthorized). "
|
||||
f"[chatgpt] Authentication failed (HTTP {response.status_code}). "
|
||||
"Your __Secure-next-auth.session-token has likely expired (~7 day lifetime). "
|
||||
"The session token is used to obtain a short-lived access token via /api/auth/session. "
|
||||
"To refresh: open chatgpt.com in Chrome → F12 → Application → Cookies "
|
||||
@@ -320,7 +323,9 @@ class ChatGPTProvider(BaseProvider):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
"authentication",
|
||||
RuntimeError("401 Unauthorized — ChatGPT token expired"),
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} — ChatGPT session token expired"
|
||||
),
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
@@ -762,14 +767,34 @@ class ChatGPTProvider(BaseProvider):
|
||||
RuntimeError(f"No download_url in response: {meta.get('detail') or meta}"),
|
||||
)
|
||||
|
||||
self._pace()
|
||||
resp = self._session.request("GET", download_url, timeout=REQUEST_TIMEOUT)
|
||||
if resp.status_code != 200:
|
||||
# The signed URL is a second hop that flakes on its own account.
|
||||
# Measured 2026-10-20 in one scheduled run: a 500 here *after* the
|
||||
# metadata hop had recovered on its 4th try, and a 503 on a different
|
||||
# asset whose metadata hop was clean first time. It therefore goes
|
||||
# through the shared retry path rather than being a bare request —
|
||||
# which also means a timeout arrives as ProviderError, so resolve_media
|
||||
# placeholders the asset, instead of as the OSError that previously
|
||||
# escaped and aborted the whole conversation.
|
||||
#
|
||||
# No _pace() here: _send_with_retries paces every attempt itself, and
|
||||
# pacing twice would add REQUEST_DELAY to every asset in the archive.
|
||||
try:
|
||||
resp = self._send_with_retries(
|
||||
"GET",
|
||||
download_url,
|
||||
max_retries=MEDIA_MAX_RETRIES,
|
||||
immediate_first_retry=True,
|
||||
time_budget=MEDIA_TIME_BUDGET,
|
||||
timeout=MEDIA_REQUEST_TIMEOUT,
|
||||
)
|
||||
except ProviderError as e:
|
||||
# Keep "which hop" in the message: that distinction is what made
|
||||
# this diagnosable, since the two hops fail for different reasons.
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"download_asset({file_id})",
|
||||
RuntimeError(f"Signed URL returned HTTP {resp.status_code}"),
|
||||
)
|
||||
RuntimeError(f"Signed URL: {e.original}"),
|
||||
) from e
|
||||
mime = resp.headers.get("content-type") or None
|
||||
file_name = meta.get("file_name") or None
|
||||
logger.debug(
|
||||
|
||||
+40
-5
@@ -2,6 +2,7 @@
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from curl_cffi import requests as curl_requests
|
||||
|
||||
@@ -38,7 +39,9 @@ class ClaudeProvider(BaseProvider):
|
||||
Cloudflare's bot detection (same issue as chatgpt.com).
|
||||
|
||||
Authentication: sessionKey cookie (~30 day lifetime, opaque string).
|
||||
Expiry cannot be decoded client-side — a 401 is the only signal.
|
||||
Expiry cannot be decoded client-side, so an API rejection is the only
|
||||
signal — and claude.ai sends 403 permission_error / account_session_invalid
|
||||
for an invalid session, not 401. See ``_is_auth_failure``.
|
||||
"""
|
||||
|
||||
provider_name = "claude"
|
||||
@@ -71,11 +74,41 @@ class ClaudeProvider(BaseProvider):
|
||||
self._org_id: str | None = None # cached per session
|
||||
logger.debug("[claude] Session initialised with Chrome TLS impersonation (key: [REDACTED])")
|
||||
|
||||
def _handle_401(self) -> None:
|
||||
# claude.ai answers an invalid or expired sessionKey with 403
|
||||
# permission_error / account_session_invalid — never 401. Verified live
|
||||
# 2026-09-20 against GET /api/organizations: a valid key returns 200, while
|
||||
# an expired key, a garbage key and no cookie at all return byte-identical
|
||||
# 403s carrying this code. Matching on the code rather than on the bare
|
||||
# status keeps a genuine permission problem (which would carry a different
|
||||
# code) reported as itself.
|
||||
_SESSION_INVALID_CODE = "account_session_invalid"
|
||||
|
||||
def _is_auth_failure(self, response: Any) -> bool:
|
||||
if response.status_code == 401:
|
||||
return True
|
||||
if response.status_code != 403:
|
||||
return False
|
||||
try:
|
||||
body = response.json()
|
||||
except Exception:
|
||||
return False
|
||||
if not isinstance(body, dict):
|
||||
return False
|
||||
error = body.get("error")
|
||||
if not isinstance(error, dict):
|
||||
return False
|
||||
details = error.get("details")
|
||||
if not isinstance(details, dict):
|
||||
return False
|
||||
return details.get("error_code") == self._SESSION_INVALID_CODE
|
||||
|
||||
def _handle_auth_failure(self, response: Any) -> None:
|
||||
msg = (
|
||||
"[claude] Authentication failed (401 Unauthorized). "
|
||||
f"[claude] Authentication failed (HTTP {response.status_code}). "
|
||||
"Your sessionKey has likely expired (~30 day lifetime). "
|
||||
"Note: Claude session keys are opaque — a 401 is the only expiry signal. "
|
||||
"Note: Claude session keys are opaque, and claude.ai reports an "
|
||||
"invalid one as 403 'Invalid authorization', not 401 — an API "
|
||||
"rejection is the only expiry signal. "
|
||||
"To refresh: open claude.ai in Chrome → F12 → Application → Cookies "
|
||||
"→ find 'sessionKey' → copy the value. "
|
||||
"Then run 'ai-chat-exporter auth' or update CLAUDE_SESSION_KEY in .env."
|
||||
@@ -84,7 +117,9 @@ class ClaudeProvider(BaseProvider):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
"authentication",
|
||||
RuntimeError("401 Unauthorized — Claude session key expired"),
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} — Claude session key expired or invalid"
|
||||
),
|
||||
)
|
||||
|
||||
def _get_org_id(self) -> str:
|
||||
|
||||
@@ -118,7 +118,10 @@ _HARNESS_TAG_RE = re.compile(
|
||||
r"|<command-args>.*?</command-args>"
|
||||
r"|<command-contents>.*?</command-contents>"
|
||||
r"|<local-command-stdout>.*?</local-command-stdout>"
|
||||
r"|<system-reminder>.*?</system-reminder>",
|
||||
r"|<system-reminder>.*?</system-reminder>"
|
||||
# A fork subagent's first user turn: generic worker rules ahead of the
|
||||
# fork's actual "Your directive: …", which is kept.
|
||||
r"|<fork-boilerplate>.*?</fork-boilerplate>",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
@@ -530,6 +533,7 @@ def _extract_messages(
|
||||
policy: str,
|
||||
subagents: dict | None = None,
|
||||
include_sidechain: bool = False,
|
||||
expanding: frozenset[str] = frozenset(),
|
||||
) -> list[dict]:
|
||||
"""Normalize Claude Code records into messages.
|
||||
|
||||
@@ -540,6 +544,13 @@ def _extract_messages(
|
||||
those recursive subagent passes (subagent records are flagged
|
||||
``isSidechain``); the top-level pass keeps skipping sidechain records so a
|
||||
subagent is never also emitted as a stray top-level turn.
|
||||
|
||||
``expanding`` holds the spawn ids of the subagents being folded around
|
||||
this pass. A ``fork`` subagent's transcript opens with a copy of the parent
|
||||
turn that spawned it, its own spawn call included; that copy is dropped,
|
||||
since the enclosing subagent block already stands for it. Expanding it
|
||||
again recursed without end (RecursionError, every daily sync from
|
||||
2026-09-24).
|
||||
"""
|
||||
subagents = subagents or {}
|
||||
messages: list[dict] = []
|
||||
@@ -620,6 +631,8 @@ def _extract_messages(
|
||||
elif item_type == "tool_use":
|
||||
name = item.get("name") or "tool"
|
||||
tool_id = item.get("id")
|
||||
if tool_id in expanding:
|
||||
continue
|
||||
if name in _SUBAGENT_TOOL_NAMES and tool_id in subagents:
|
||||
# A Task/Agent spawn: fold its separate transcript inline
|
||||
# instead of collapsing it. Its own tool traffic is
|
||||
@@ -633,6 +646,7 @@ def _extract_messages(
|
||||
policy,
|
||||
subagents,
|
||||
include_sidechain=True,
|
||||
expanding=expanding | {tool_id},
|
||||
)
|
||||
blocks.append(
|
||||
make_subagent_block(
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Shared test fixtures.
|
||||
|
||||
The autouse fixture here exists because of a real incident (2026-08-18): the
|
||||
`sync` CLI tests invoke the actual command, which calls `load_config()`, which
|
||||
calls `load_dotenv()` — so the developer's real `.env` was loaded and its
|
||||
`NTFY_TOPIC` used. Every `pytest` run fired real push notifications at the
|
||||
developer's phone, including a fabricated "3 conversations failed to export"
|
||||
from a fixture. Nothing appeared in the log to explain it, because the tests
|
||||
pass `--no-log-file`.
|
||||
|
||||
The lesson generalises past ntfy: any test that exercises a command end to end
|
||||
inherits whatever is in `.env` unless the environment is neutralised first.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_outbound_side_effects(monkeypatch):
|
||||
"""Neutralise every environment variable that could reach a real service.
|
||||
|
||||
Set to empty/unroutable values rather than deleted: `load_dotenv` is called
|
||||
with ``override=False``, which only skips keys **already present** in the
|
||||
environment. Deleting a key would let the developer's `.env` put it back.
|
||||
|
||||
Individual tests may still `monkeypatch.setenv` these — that is how the
|
||||
notification tests point at a dead local port on purpose.
|
||||
"""
|
||||
# Notifications: an empty topic disables sending outright (`is_configured`
|
||||
# strips and checks truthiness), and the server is pointed at a closed port
|
||||
# so even a test that sets its own topic cannot reach the internet.
|
||||
monkeypatch.setenv("NTFY_TOPIC", "")
|
||||
monkeypatch.setenv("NTFY_SERVER", "http://127.0.0.1:9")
|
||||
monkeypatch.setenv("NTFY_TOKEN", "")
|
||||
|
||||
# Joplin: a test that reached the developer's running desktop instance would
|
||||
# create or overwrite real notes in their archive.
|
||||
monkeypatch.setenv("JOPLIN_API_URL", "http://127.0.0.1:9")
|
||||
monkeypatch.setenv("JOPLIN_API_TOKEN", "")
|
||||
@@ -254,3 +254,97 @@ class TestReexportPreservesJoplinLink:
|
||||
pending = tmp_cache.get_joplin_pending("chatgpt")
|
||||
assert len(pending) == 1
|
||||
assert pending[0][1]["joplin_note_id"] == "note-123"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cross-run media retry.
|
||||
#
|
||||
# Before this, a transient asset failure was permanent: resolve_media kept a
|
||||
# placeholder, mark_exported ran anyway, and the conversation was never looked
|
||||
# at again. Two assets were lost that way on 2026-10-20 to a 500 and a 503 —
|
||||
# both recoverable. The manifest now remembers which refs are still missing.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestPendingMediaRetry:
|
||||
UPDATED = "2026-10-20T09:00:00Z"
|
||||
|
||||
def test_pending_media_forces_a_re_export(self, tmp_cache):
|
||||
tmp_cache.mark_exported(
|
||||
"chatgpt", "conv-1",
|
||||
{"updated_at": self.UPDATED, "pending_media": ["sediment://file_x"]},
|
||||
)
|
||||
assert tmp_cache.is_cached("chatgpt", "conv-1", self.UPDATED) is False
|
||||
|
||||
def test_no_pending_media_is_an_ordinary_hit(self, tmp_cache):
|
||||
tmp_cache.mark_exported("chatgpt", "conv-1", {"updated_at": self.UPDATED})
|
||||
assert tmp_cache.is_cached("chatgpt", "conv-1", self.UPDATED) is True
|
||||
|
||||
def test_a_resolved_asset_clears_the_flag(self, tmp_cache):
|
||||
"""The run that recovers the asset must stop the daily re-export."""
|
||||
tmp_cache.mark_exported(
|
||||
"chatgpt", "conv-1",
|
||||
{"updated_at": self.UPDATED, "pending_media": ["sediment://file_x"]},
|
||||
)
|
||||
assert tmp_cache.is_cached("chatgpt", "conv-1", self.UPDATED) is False
|
||||
|
||||
tmp_cache.mark_exported("chatgpt", "conv-1", {"updated_at": self.UPDATED})
|
||||
assert tmp_cache.is_cached("chatgpt", "conv-1", self.UPDATED) is True
|
||||
entry = tmp_cache.get_all_entries("chatgpt")["conv-1"]
|
||||
assert "pending_media" not in entry
|
||||
|
||||
def test_retries_are_capped_so_it_does_not_re_export_forever(self, tmp_cache):
|
||||
"""An asset that 5xxs every morning must stop costing a re-export."""
|
||||
from src.cache import MAX_MEDIA_RETRY_RUNS
|
||||
|
||||
for run in range(MAX_MEDIA_RETRY_RUNS):
|
||||
tmp_cache.mark_exported(
|
||||
"chatgpt", "conv-1",
|
||||
{"updated_at": self.UPDATED, "pending_media": ["sediment://file_x"]},
|
||||
)
|
||||
assert tmp_cache.is_cached("chatgpt", "conv-1", self.UPDATED) is False, (
|
||||
f"run {run + 1} should still retry"
|
||||
)
|
||||
|
||||
tmp_cache.mark_exported(
|
||||
"chatgpt", "conv-1",
|
||||
{"updated_at": self.UPDATED, "pending_media": ["sediment://file_x"]},
|
||||
)
|
||||
assert tmp_cache.is_cached("chatgpt", "conv-1", self.UPDATED) is True
|
||||
|
||||
def test_the_lost_refs_are_kept_as_a_record(self, tmp_cache):
|
||||
from src.cache import MAX_MEDIA_RETRY_RUNS
|
||||
|
||||
for _ in range(MAX_MEDIA_RETRY_RUNS + 1):
|
||||
tmp_cache.mark_exported(
|
||||
"chatgpt", "conv-1",
|
||||
{"updated_at": self.UPDATED, "pending_media": ["sediment://file_x"]},
|
||||
)
|
||||
entry = tmp_cache.get_all_entries("chatgpt")["conv-1"]
|
||||
assert entry["pending_media"] == ["sediment://file_x"]
|
||||
|
||||
def test_joplin_link_survives_a_media_re_export(self, tmp_cache):
|
||||
"""Re-exporting to recover an asset must not orphan the Joplin note."""
|
||||
tmp_cache.mark_exported("chatgpt", "conv-1", {"updated_at": self.UPDATED})
|
||||
tmp_cache.mark_joplin_synced("chatgpt", "conv-1", "note-123")
|
||||
tmp_cache.set_joplin_resources("chatgpt", "conv-1", {"media/a.png": "res-1"})
|
||||
|
||||
tmp_cache.mark_exported(
|
||||
"chatgpt", "conv-1",
|
||||
{"updated_at": self.UPDATED, "pending_media": ["sediment://file_x"]},
|
||||
)
|
||||
entry = tmp_cache.get_all_entries("chatgpt")["conv-1"]
|
||||
assert entry["joplin_note_id"] == "note-123"
|
||||
# Resource ids are the idempotency record: without them the re-sync
|
||||
# would re-upload every image in the note, not just the new one.
|
||||
assert entry["joplin_resources"] == {"media/a.png": "res-1"}
|
||||
|
||||
def test_pending_media_survives_a_manifest_round_trip(self, tmp_path):
|
||||
from src.cache import Cache
|
||||
|
||||
c = Cache(tmp_path)
|
||||
c.mark_exported(
|
||||
"chatgpt", "conv-1",
|
||||
{"updated_at": self.UPDATED, "pending_media": ["sediment://file_x"]},
|
||||
)
|
||||
assert Cache(tmp_path).is_cached("chatgpt", "conv-1", self.UPDATED) is False
|
||||
|
||||
@@ -370,6 +370,59 @@ class TestSubagentFold:
|
||||
]
|
||||
assert "stray sidechain" not in top_text
|
||||
|
||||
def test_fork_containing_its_own_spawn_call(self, tmp_path):
|
||||
# The shape Claude Code writes for a `fork` subagent: a context-ref
|
||||
# record, a copy of the parent turn holding the fork's own spawn call,
|
||||
# then the directive behind <fork-boilerplate>. Folding that copied
|
||||
# call recursed until RecursionError.
|
||||
fork_records = [
|
||||
{"type": "fork-context-ref", "agentId": "a1", "parentLastUuid": "u0"},
|
||||
{
|
||||
"type": "assistant", "isSidechain": True,
|
||||
"message": {"role": "assistant", "content": [
|
||||
{"type": "tool_use", "id": "toolu_sub1", "name": "Agent",
|
||||
"input": {"description": "research"}},
|
||||
]},
|
||||
},
|
||||
{
|
||||
"type": "user", "isSidechain": True,
|
||||
"timestamp": "2026-05-01T10:00:06.000Z",
|
||||
"message": {"role": "user", "content": [
|
||||
{"type": "tool_result", "tool_use_id": "toolu_sub1",
|
||||
"content": [{"type": "text", "text": "Fork started"}]},
|
||||
{"type": "text", "text":
|
||||
"<fork-boilerplate>\nYou are a worker fork.\n"
|
||||
"</fork-boilerplate>\n\nYour directive: research X"},
|
||||
]},
|
||||
},
|
||||
{
|
||||
"type": "assistant", "isSidechain": True,
|
||||
"timestamp": "2026-05-01T10:00:07.000Z",
|
||||
"message": {"role": "assistant", "content": [
|
||||
{"type": "text", "text": "Fork result."},
|
||||
]},
|
||||
},
|
||||
]
|
||||
f = _write_session(tmp_path, _parent_with_task(), name="parent-3")
|
||||
_write_subagent(
|
||||
f, "toolu_sub1", fork_records,
|
||||
agent_type="fork", description="research X",
|
||||
)
|
||||
p = self._provider(tmp_path)
|
||||
p.fetch_all_conversations()
|
||||
conv = p.normalize_conversation(p.get_conversation("parent-3"))
|
||||
|
||||
subs = [
|
||||
b for m in conv["messages"] for b in m["blocks"] if b["type"] == "subagent"
|
||||
]
|
||||
assert len(subs) == 1
|
||||
inner = [bb for m in subs[0]["messages"] for bb in m["blocks"]]
|
||||
assert not any(b["type"] == "subagent" for b in inner)
|
||||
inner_text = [b.get("text") for b in inner]
|
||||
assert "Your directive: research X" in inner_text
|
||||
assert "Fork result." in inner_text
|
||||
assert not any("worker fork" in (t or "") for t in inner_text)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Repo tags in title (Part C)
|
||||
|
||||
@@ -624,3 +624,103 @@ class TestNotifySend:
|
||||
monkeypatch.setenv("NTFY_TOPIC", "unit-test-topic")
|
||||
monkeypatch.setenv("NTFY_NOTIFY", "off")
|
||||
assert notify_mod.is_configured() is False
|
||||
|
||||
|
||||
class TestNoRealNotificationsDuringTests:
|
||||
"""Regression guard for the 2026-08-18 incident: the suite pushed to the
|
||||
developer's real ntfy topic because `sync` loads `.env` via `load_dotenv`.
|
||||
|
||||
Asserts the conftest neutralisation holds even though a real `.env` with a
|
||||
live NTFY_TOPIC sits beside the tests.
|
||||
"""
|
||||
|
||||
def test_notifications_are_disabled(self):
|
||||
from src import notify as notify_mod
|
||||
|
||||
assert notify_mod.is_configured() is False
|
||||
|
||||
def test_dotenv_cannot_reintroduce_a_topic(self):
|
||||
"""`load_dotenv(override=False)` skips keys already present — including
|
||||
empty ones. Deleting the var instead of emptying it would reopen this."""
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from src import notify as notify_mod
|
||||
|
||||
load_dotenv(override=False)
|
||||
assert os.getenv("NTFY_TOPIC", "").strip() == ""
|
||||
assert notify_mod.is_configured() is False
|
||||
|
||||
def test_send_cannot_reach_the_public_server(self, monkeypatch):
|
||||
"""Even a test that sets its own topic is pinned to a dead local port."""
|
||||
import os
|
||||
|
||||
from src import notify as notify_mod
|
||||
|
||||
monkeypatch.setenv("NTFY_TOPIC", "some-topic")
|
||||
assert "127.0.0.1" in os.getenv("NTFY_SERVER", "")
|
||||
assert notify_mod.send("t", "m") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Media failures in the push notification.
|
||||
#
|
||||
# On 2026-10-20 a scheduled run lost two assets and pushed a clean "OK" with
|
||||
# no hint of it — the counts were in the log file only, which is a place you
|
||||
# have to think to go and look. Counts only, never asset ids: a topic on
|
||||
# public ntfy.sh is readable by anyone who knows its name.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestMediaFailuresInNotification:
|
||||
def test_media_failures_appear_in_the_message(self):
|
||||
from src.notify import format_summary
|
||||
|
||||
_, message, _, _ = format_summary(
|
||||
{"chatgpt": {"exported": 24, "skipped": 826, "failed": 0, "media_failed": 2}},
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert "2 media failed" in message
|
||||
|
||||
def test_absent_when_there_are_none(self):
|
||||
from src.notify import format_summary
|
||||
|
||||
_, message, _, _ = format_summary(
|
||||
{"chatgpt": {"exported": 5, "skipped": 1, "failed": 0, "media_failed": 0}},
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert "media failed" not in message
|
||||
|
||||
def test_media_failures_alone_do_not_fail_the_run(self):
|
||||
"""The transcript was archived and a transient asset retries later."""
|
||||
from src.notify import format_summary
|
||||
|
||||
title, _, tags, priority = format_summary(
|
||||
{"chatgpt": {"exported": 1, "skipped": 0, "failed": 0, "media_failed": 3}},
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert title.startswith("AI archive OK")
|
||||
assert priority == "default"
|
||||
assert tags == "white_check_mark"
|
||||
|
||||
def test_no_asset_ids_leak_into_the_payload(self):
|
||||
from src.notify import format_summary
|
||||
|
||||
_, message, _, _ = format_summary(
|
||||
{"chatgpt": {"exported": 1, "skipped": 0, "failed": 0, "media_failed": 1}},
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert "sediment" not in message and "file_" not in message
|
||||
|
||||
def test_older_summaries_without_the_key_still_render(self):
|
||||
from src.notify import format_summary
|
||||
|
||||
_, message, _, _ = format_summary(
|
||||
{"codex": {"exported": 0, "skipped": 29, "failed": 0}}, None, None
|
||||
)
|
||||
assert "codex: 0 exported, 29 up to date" in message
|
||||
|
||||
+163
-6
@@ -100,7 +100,7 @@ class TestResolveMedia:
|
||||
|
||||
n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
|
||||
|
||||
assert n == 1
|
||||
assert n.downloaded == 1
|
||||
assert report.media_downloaded == 1
|
||||
assert block["local_path"] == "media/file_img1.png"
|
||||
rendered = render_blocks_to_markdown([block])
|
||||
@@ -117,7 +117,7 @@ class TestResolveMedia:
|
||||
report = LossReport()
|
||||
|
||||
n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
|
||||
assert n == 0
|
||||
assert n.downloaded == 0
|
||||
assert "local_path" not in block
|
||||
assert provider.calls == []
|
||||
|
||||
@@ -129,7 +129,7 @@ class TestResolveMedia:
|
||||
report = LossReport()
|
||||
|
||||
n = resolve_media(conv, provider, tmp_path, "provider/project/year", "all", report)
|
||||
assert n == 1
|
||||
assert n.downloaded == 1
|
||||
assert block["local_path"] == "media/file_audio1.wav"
|
||||
rendered = render_blocks_to_markdown([block])
|
||||
assert "media/file_audio1.wav" in rendered and rendered.startswith("> 📎")
|
||||
@@ -139,7 +139,9 @@ class TestResolveMedia:
|
||||
block = make_image_placeholder(ref="sediment://file_x", source="user_upload")
|
||||
conv = _conv_with([block])
|
||||
report = LossReport()
|
||||
assert resolve_media(conv, provider, tmp_path, "provider/project/year", "off", report) == 0
|
||||
assert resolve_media(
|
||||
conv, provider, tmp_path, "provider/project/year", "off", report
|
||||
).downloaded == 0
|
||||
assert provider.calls == []
|
||||
|
||||
def test_idempotent_uses_disk(self, tmp_path):
|
||||
@@ -164,7 +166,7 @@ class TestResolveMedia:
|
||||
report = LossReport()
|
||||
|
||||
n = resolve_media(conv, provider, tmp_path, "provider/project/year", "images", report)
|
||||
assert n == 0
|
||||
assert n.downloaded == 0
|
||||
assert "local_path" not in block
|
||||
assert report.media_failed["expired-or-missing"] == 1
|
||||
# Still renders as a placeholder, not a broken image link
|
||||
@@ -192,7 +194,9 @@ class TestResolveMedia:
|
||||
pass
|
||||
block = make_image_placeholder(ref="sediment://file_x", source="user_upload")
|
||||
conv = _conv_with([block])
|
||||
assert resolve_media(conv, NoDownload(), tmp_path, "provider/project/year", "images", LossReport()) == 0
|
||||
assert resolve_media(
|
||||
conv, NoDownload(), tmp_path, "provider/project/year", "images", LossReport()
|
||||
).downloaded == 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -262,3 +266,156 @@ class TestUploadMediaAndRewrite:
|
||||
new_body, res_map = upload_media_and_rewrite(body, tmp_path, client, {})
|
||||
assert new_body == body
|
||||
assert res_map == {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Transient vs permanent failures, and the cross-run retry they feed.
|
||||
#
|
||||
# Measured 2026-10-20 in one scheduled run: a signed URL returned 500 on one
|
||||
# asset and 503 on another. Both are recoverable, yet the conversation was
|
||||
# marked exported, so the assets were lost for good. Permanent failures must
|
||||
# stay out of the retry set — 19 assets in this archive 403/404 because their
|
||||
# bytes were lost in a 2026-07-14 Library migration, and retrying those every
|
||||
# morning would be noise that also trips the circuit breaker.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestTransientFailureRetry:
|
||||
def _run(self, failure, tmp_path, circuit=None):
|
||||
from src.media import resolve_media
|
||||
|
||||
ref = "sediment://file_x"
|
||||
provider = _FakeProvider(fail_refs={ref: RuntimeError(failure)})
|
||||
conv = _conv_with([make_image_placeholder(ref, "user_upload")])
|
||||
report = LossReport()
|
||||
outcome = resolve_media(
|
||||
conv, provider, tmp_path, "provider/project/year", "images", report,
|
||||
circuit=circuit,
|
||||
)
|
||||
return outcome, report, conv
|
||||
|
||||
def test_server_error_is_retryable(self, tmp_path):
|
||||
outcome, report, conv = self._run("Signed URL: HTTP 500 after 5 retries", tmp_path)
|
||||
assert outcome.failed == 1
|
||||
assert outcome.retryable_refs == ["sediment://file_x"]
|
||||
assert report.media_failed["download-error"] == 1
|
||||
# Placeholder survives, so the ref is still in the Markdown.
|
||||
assert "local_path" not in conv["messages"][0]["blocks"][0]
|
||||
|
||||
def test_service_unavailable_is_retryable(self, tmp_path):
|
||||
outcome, _, _ = self._run("Signed URL: HTTP 503 after 5 retries", tmp_path)
|
||||
assert outcome.retryable_refs == ["sediment://file_x"]
|
||||
|
||||
def test_timeout_is_retryable(self, tmp_path):
|
||||
outcome, _, _ = self._run(
|
||||
"Network error after 5 retries: Operation timed out", tmp_path
|
||||
)
|
||||
assert outcome.retryable_refs == ["sediment://file_x"]
|
||||
|
||||
def test_deleted_asset_is_not_retryable(self, tmp_path):
|
||||
outcome, report, _ = self._run(
|
||||
"Asset no longer exists — HTTP 404 'File not found'", tmp_path
|
||||
)
|
||||
assert outcome.failed == 1
|
||||
assert outcome.retryable_refs == []
|
||||
assert report.media_failed["expired-or-missing"] == 1
|
||||
|
||||
def test_forbidden_asset_is_not_retryable(self, tmp_path):
|
||||
outcome, report, _ = self._run("HTTP 403 — detail: Forbidden", tmp_path)
|
||||
assert outcome.retryable_refs == []
|
||||
assert report.media_failed["forbidden"] == 1
|
||||
|
||||
def test_success_reports_nothing_retryable(self, tmp_path):
|
||||
from src.media import resolve_media
|
||||
|
||||
ref = "sediment://file_x"
|
||||
provider = _FakeProvider(assets={ref: (b"png", "image/png", None)})
|
||||
conv = _conv_with([make_image_placeholder(ref, "user_upload")])
|
||||
outcome = resolve_media(
|
||||
conv, provider, tmp_path, "provider/project/year", "images", LossReport()
|
||||
)
|
||||
assert (outcome.downloaded, outcome.failed, outcome.retryable_refs) == (1, 0, [])
|
||||
|
||||
|
||||
class TestMediaCircuit:
|
||||
def test_trips_after_consecutive_transient_failures(self):
|
||||
from src.media import MediaCircuit
|
||||
|
||||
c = MediaCircuit(threshold=3)
|
||||
for _ in range(2):
|
||||
c.record_failure("download-error")
|
||||
assert not c.tripped
|
||||
c.record_failure("download-error")
|
||||
assert c.tripped
|
||||
|
||||
def test_success_resets_the_streak(self):
|
||||
from src.media import MediaCircuit
|
||||
|
||||
c = MediaCircuit(threshold=3)
|
||||
c.record_failure("download-error")
|
||||
c.record_failure("download-error")
|
||||
c.record_success()
|
||||
c.record_failure("download-error")
|
||||
assert not c.tripped, "scattered failures must not trip the breaker"
|
||||
|
||||
def test_permanent_failures_never_trip_it(self):
|
||||
"""A deleted upload says nothing about the provider's health."""
|
||||
from src.media import MediaCircuit
|
||||
|
||||
c = MediaCircuit(threshold=2)
|
||||
for _ in range(10):
|
||||
c.record_failure("expired-or-missing")
|
||||
c.record_failure("forbidden")
|
||||
assert not c.tripped
|
||||
|
||||
def test_open_circuit_skips_downloads_but_keeps_refs(self, tmp_path):
|
||||
from src.media import CIRCUIT_OPEN_BUCKET, MediaCircuit, resolve_media
|
||||
|
||||
ref = "sediment://file_y"
|
||||
provider = _FakeProvider(assets={ref: (b"png", "image/png", None)})
|
||||
conv = _conv_with([make_image_placeholder(ref, "user_upload")])
|
||||
circuit = MediaCircuit(threshold=1)
|
||||
circuit.record_failure("download-error")
|
||||
assert circuit.tripped
|
||||
|
||||
report = LossReport()
|
||||
outcome = resolve_media(
|
||||
conv, provider, tmp_path, "provider/project/year", "images", report,
|
||||
circuit=circuit,
|
||||
)
|
||||
assert provider.calls == [], "must not touch the network while open"
|
||||
assert outcome.downloaded == 0
|
||||
assert outcome.retryable_refs == [ref], "skipped assets must not be abandoned"
|
||||
assert report.media_failed[CIRCUIT_OPEN_BUCKET] == 1
|
||||
assert circuit.skipped == 1
|
||||
|
||||
def test_already_downloaded_asset_served_while_open(self, tmp_path):
|
||||
"""An open circuit must not stop on-disk media being linked."""
|
||||
from src.media import MediaCircuit, resolve_media
|
||||
|
||||
ref = "sediment://file_z"
|
||||
conv = _conv_with([make_image_placeholder(ref, "user_upload")])
|
||||
# Derive the directory the way the exporter does rather than
|
||||
# hardcoding it — the layout is "<provider>/<project>.<year>/".
|
||||
from src.utils import build_export_path, generate_filename
|
||||
|
||||
media_dir = build_export_path(
|
||||
tmp_path,
|
||||
"chatgpt",
|
||||
None,
|
||||
"2026-05-20T00:00:00+00:00",
|
||||
generate_filename("Has Media", "conv-1", "2026-05-20T00:00:00+00:00"),
|
||||
"provider/project/year",
|
||||
).parent / "media"
|
||||
media_dir.mkdir(parents=True)
|
||||
(media_dir / "file_z.png").write_bytes(b"already here")
|
||||
|
||||
circuit = MediaCircuit(threshold=1)
|
||||
circuit.record_failure("download-error")
|
||||
provider = _FakeProvider()
|
||||
resolve_media(
|
||||
conv, provider, tmp_path, "provider/project/year", "images", LossReport(),
|
||||
circuit=circuit,
|
||||
)
|
||||
assert provider.calls == []
|
||||
assert conv["messages"][0]["blocks"][0]["local_path"] == "media/file_z.png"
|
||||
|
||||
@@ -1184,6 +1184,133 @@ class TestErrorBodyDiagnostics:
|
||||
assert "[REDACTED]" in described
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auth failure detection: claude.ai reports an invalid/expired sessionKey as
|
||||
# 403 permission_error / account_session_invalid, never 401. Verified live
|
||||
# 2026-09-20 against GET /api/organizations — a valid key returned 200, while
|
||||
# an expired key, a garbage key and no cookie at all returned byte-identical
|
||||
# 403s. Keying auth handling on 401 alone hid the refresh instructions behind
|
||||
# a raw JSON dump.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAuthFailureDetection:
|
||||
class _Resp:
|
||||
reason = ""
|
||||
headers: dict = {}
|
||||
|
||||
def __init__(self, status, payload=None):
|
||||
self.status_code = status
|
||||
self.ok = 200 <= status < 400
|
||||
self._payload = payload
|
||||
self.text = ""
|
||||
|
||||
def json(self):
|
||||
if self._payload is None:
|
||||
raise ValueError("not json")
|
||||
return self._payload
|
||||
|
||||
@staticmethod
|
||||
def _bare(cls, response):
|
||||
p = cls.__new__(cls)
|
||||
p._request_delay = 0
|
||||
p._last_request_at = None
|
||||
p._session = type("S", (), {"request": lambda *a, **k: response})()
|
||||
return p
|
||||
|
||||
@staticmethod
|
||||
def _session_invalid(message="Invalid authorization"):
|
||||
return {
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "permission_error",
|
||||
"message": message,
|
||||
"details": {
|
||||
"error_code": "account_session_invalid",
|
||||
"error_visibility": "user_facing",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
def test_claude_403_session_invalid_is_an_auth_failure(self):
|
||||
from src.providers.base import ProviderError
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
resp = self._Resp(403, self._session_invalid())
|
||||
prov = self._bare(ClaudeProvider, resp)
|
||||
assert prov._is_auth_failure(resp) is True
|
||||
|
||||
with pytest.raises(ProviderError) as exc:
|
||||
prov._make_request("GET", "https://claude.ai/api/organizations")
|
||||
# The actionable message, not a dump of the response body.
|
||||
assert exc.value.operation == "authentication"
|
||||
assert "session key expired or invalid" in str(exc.value.original)
|
||||
|
||||
def test_claude_403_auth_failure_tells_the_user_how_to_refresh(self, caplog):
|
||||
from src.providers.base import ProviderError
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
resp = self._Resp(403, self._session_invalid())
|
||||
with caplog.at_level(logging.ERROR):
|
||||
with pytest.raises(ProviderError):
|
||||
self._bare(ClaudeProvider, resp)._make_request(
|
||||
"GET", "https://claude.ai/api/organizations"
|
||||
)
|
||||
logged = caplog.text
|
||||
assert "sessionKey" in logged
|
||||
assert "CLAUDE_SESSION_KEY" in logged
|
||||
# States the status it actually saw, rather than claiming 401.
|
||||
assert "403" in logged
|
||||
assert "401 Unauthorized" not in logged
|
||||
|
||||
def test_claude_403_with_another_error_code_is_not_an_auth_failure(self):
|
||||
"""A genuine permission problem must stay reported as itself."""
|
||||
from src.providers.base import ProviderError
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
payload = {
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "permission_error",
|
||||
"message": "Organization access denied",
|
||||
"details": {"error_code": "org_access_denied"},
|
||||
},
|
||||
}
|
||||
resp = self._Resp(403, payload)
|
||||
prov = self._bare(ClaudeProvider, resp)
|
||||
assert prov._is_auth_failure(resp) is False
|
||||
|
||||
with pytest.raises(ProviderError) as exc:
|
||||
prov._make_request("GET", "https://claude.ai/api/organizations")
|
||||
assert exc.value.operation != "authentication"
|
||||
assert "Organization access denied" in str(exc.value.original)
|
||||
|
||||
def test_claude_403_with_unparseable_body_is_not_an_auth_failure(self):
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
resp = self._Resp(403, None)
|
||||
assert self._bare(ClaudeProvider, resp)._is_auth_failure(resp) is False
|
||||
|
||||
def test_claude_401_is_still_an_auth_failure(self):
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
resp = self._Resp(401, {})
|
||||
assert self._bare(ClaudeProvider, resp)._is_auth_failure(resp) is True
|
||||
|
||||
def test_chatgpt_403_is_not_an_auth_failure(self):
|
||||
"""Guards the deleted-asset path: a media 403 is not an expired token."""
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
|
||||
resp = self._Resp(403, {"detail": "Forbidden"})
|
||||
assert self._bare(ChatGPTProvider, resp)._is_auth_failure(resp) is False
|
||||
|
||||
def test_chatgpt_401_is_an_auth_failure(self):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
|
||||
resp = self._Resp(401, {})
|
||||
assert self._bare(ChatGPTProvider, resp)._is_auth_failure(resp) is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deleted assets: ChatGPT's download endpoint answers a missing upload with
|
||||
# 403 Forbidden, not 404. Measured 2026-08-17 over 18 such assets — every one
|
||||
@@ -1272,6 +1399,8 @@ class TestDeletedAssetReporting:
|
||||
"""The probe costs a request — it must not fire on a good download."""
|
||||
class _Bytes:
|
||||
status_code = 200
|
||||
ok = True
|
||||
reason = "OK"
|
||||
content = b"data"
|
||||
headers = {"content-type": "image/png"}
|
||||
|
||||
@@ -1379,3 +1508,236 @@ class TestProjectAttribution:
|
||||
with caplog.at_level(logging.INFO):
|
||||
p.normalize_conversation(self._raw(gizmo_id="g-p-known"))
|
||||
assert not [r for r in caplog.records if "not in CHATGPT_PROJECT_IDS" in r.message]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Asset download retries. Measured 2026-10-20 in one scheduled run:
|
||||
#
|
||||
# 09:04:05 .../files/<id>/download → 500 Retry 1/3 in 2.2s
|
||||
# 09:04:17 → 500 Retry 2/3 in 4.3s
|
||||
# 09:04:32 → 500 Retry 3/3 in 9.1s
|
||||
# 09:04:42 → 200 ← metadata hop recovered
|
||||
# 09:04:47 WARNING Signed URL returned HTTP 500 ← second hop, no retry
|
||||
#
|
||||
# The metadata hop went through _make_request and won; the signed-URL hop was
|
||||
# a bare request and lost on its first try. A second asset lost the same way
|
||||
# to a 503, and a third timed out — and because curl_cffi's RequestException
|
||||
# subclasses OSError, that timeout escaped resolve_media's `except
|
||||
# ProviderError` and was reported by the export loop as "File write failed",
|
||||
# aborting a conversation that had already downloaded ~20 images.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAssetDownloadRetries:
|
||||
class _Resp:
|
||||
def __init__(self, status=200, payload=None, content=b"", headers=None):
|
||||
self.status_code = status
|
||||
self.ok = 200 <= status < 400
|
||||
self.reason = ""
|
||||
self.content = content
|
||||
self.text = ""
|
||||
self.headers = headers or {}
|
||||
self._payload = payload
|
||||
|
||||
def json(self):
|
||||
if self._payload is None:
|
||||
raise ValueError("not json")
|
||||
return self._payload
|
||||
|
||||
class _ScriptedSession:
|
||||
"""Returns scripted responses in order; an Exception entry is raised."""
|
||||
|
||||
def __init__(self, script):
|
||||
self.script = list(script)
|
||||
self.requests = []
|
||||
self.headers = {}
|
||||
|
||||
def request(self, method, url, **kwargs):
|
||||
self.requests.append((method, url, kwargs))
|
||||
item = self.script.pop(0)
|
||||
if isinstance(item, BaseException):
|
||||
raise item
|
||||
return item
|
||||
|
||||
def _provider(self, script):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
|
||||
p = ChatGPTProvider.__new__(ChatGPTProvider)
|
||||
p._request_delay = 0
|
||||
p._last_request_at = None
|
||||
p._session = self._ScriptedSession(script)
|
||||
return p
|
||||
|
||||
def _meta(self):
|
||||
"""Hop 1: metadata response carrying the signed download_url."""
|
||||
return self._Resp(payload={"download_url": "https://files.example/signed"})
|
||||
|
||||
def test_signed_url_5xx_is_retried_and_recovers(self, monkeypatch):
|
||||
"""The exact 2026-10-20 failure: hop 2 500s once, then succeeds."""
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
p = self._provider([
|
||||
self._meta(),
|
||||
self._Resp(500),
|
||||
self._Resp(200, content=b"PNGDATA", headers={"content-type": "image/png"}),
|
||||
])
|
||||
content, mime, _ = p.download_asset("sediment://file_x")
|
||||
assert content == b"PNGDATA"
|
||||
assert mime == "image/png"
|
||||
assert len(p._session.requests) == 3, "hop 2 must be retried, not abandoned"
|
||||
|
||||
def test_signed_url_503_is_retried(self, monkeypatch):
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
p = self._provider([
|
||||
self._meta(),
|
||||
self._Resp(503),
|
||||
self._Resp(200, content=b"OK", headers={"content-type": "image/png"}),
|
||||
])
|
||||
assert p.download_asset("sediment://file_x")[0] == b"OK"
|
||||
|
||||
def test_signed_url_timeout_raises_provider_error_not_oserror(self, monkeypatch):
|
||||
"""The conversation-killer: a hop-2 timeout must be a ProviderError.
|
||||
|
||||
curl_cffi's RequestException is an OSError subclass, so an un-wrapped
|
||||
timeout here bypassed resolve_media's handler and was caught by the
|
||||
export loop's `except OSError` as "File write failed".
|
||||
"""
|
||||
from curl_cffi.requests.exceptions import Timeout
|
||||
from src.providers.base import ProviderError
|
||||
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
assert issubclass(Timeout, OSError), "premise of this test"
|
||||
|
||||
p = self._provider([self._meta()] + [Timeout("timed out")] * 8)
|
||||
with pytest.raises(ProviderError) as exc:
|
||||
p.download_asset("sediment://file_x")
|
||||
assert "Signed URL" in str(exc.value.original), "must name the failing hop"
|
||||
|
||||
def test_which_hop_failed_stays_in_the_message(self, monkeypatch):
|
||||
"""Hop 1 vs hop 2 fail for different reasons; the message must say which."""
|
||||
from src.providers.base import ProviderError
|
||||
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
p = self._provider([self._meta()] + [self._Resp(500)] * 8)
|
||||
with pytest.raises(ProviderError) as exc:
|
||||
p.download_asset("sediment://file_x")
|
||||
detail = str(exc.value.original)
|
||||
assert detail.startswith("Signed URL:")
|
||||
assert "500" in detail
|
||||
|
||||
def test_hop_two_is_paced_once_per_attempt_not_twice(self, monkeypatch):
|
||||
"""Pacing twice per asset would add REQUEST_DELAY to every image."""
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
p = self._provider([
|
||||
self._meta(),
|
||||
self._Resp(200, content=b"X", headers={"content-type": "image/png"}),
|
||||
])
|
||||
calls = []
|
||||
real_pace = p._pace
|
||||
monkeypatch.setattr(p, "_pace", lambda: calls.append(1) or real_pace())
|
||||
p.download_asset("sediment://file_x")
|
||||
assert len(calls) == len(p._session.requests) == 2
|
||||
|
||||
|
||||
class TestRetryLadder:
|
||||
def test_immediate_first_retry_costs_nothing(self):
|
||||
from src.providers.base import BaseProvider
|
||||
|
||||
assert BaseProvider._backoff_wait(None, 1, immediate_first=True) == 0.0
|
||||
|
||||
def test_ladder_shifts_down_so_it_reads_0_2_4_8_16(self):
|
||||
from src.providers.base import BaseProvider
|
||||
|
||||
for attempt, lo in ((2, 2), (3, 4), (4, 8), (5, 16)):
|
||||
wait = BaseProvider._backoff_wait(None, attempt, immediate_first=True)
|
||||
assert lo <= wait <= lo * 1.3, f"attempt {attempt} → {wait}"
|
||||
|
||||
def test_jitter_is_never_dropped(self):
|
||||
"""A storage incident fails many assets; un-jittered retries sync up."""
|
||||
from src.providers.base import BaseProvider
|
||||
|
||||
waits = {
|
||||
BaseProvider._backoff_wait(None, 3, immediate_first=True)
|
||||
for _ in range(25)
|
||||
}
|
||||
assert len(waits) > 1
|
||||
|
||||
def test_conversation_api_ladder_is_unchanged(self):
|
||||
from src.providers.base import BaseProvider
|
||||
|
||||
assert 2 <= BaseProvider._backoff_wait(None, 1) <= 2.6
|
||||
assert 4 <= BaseProvider._backoff_wait(None, 2) <= 5.2
|
||||
|
||||
def test_retry_after_wins_over_the_immediate_retry(self):
|
||||
"""A server naming a time knows better than we do; 503s often do."""
|
||||
from src.providers.base import BaseProvider
|
||||
|
||||
class _R:
|
||||
headers = {"Retry-After": "7"}
|
||||
|
||||
assert BaseProvider._backoff_wait(_R(), 1, immediate_first=True) == 7.0
|
||||
|
||||
|
||||
class TestMediaTimeBudget:
|
||||
def test_budget_refuses_a_wait_it_cannot_afford(self, monkeypatch):
|
||||
"""Bounds the timeout case, which costs 40s a go and has no cheap end."""
|
||||
from src.providers.base import ProviderError
|
||||
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
p = TestAssetDownloadRetries()._provider([
|
||||
TestAssetDownloadRetries._Resp(500) for _ in range(9)
|
||||
])
|
||||
# First backoff is 2-2.6s, which does not fit in a 1s budget.
|
||||
with pytest.raises(ProviderError):
|
||||
p._send_with_retries(
|
||||
"GET", "https://files.example/signed",
|
||||
max_retries=5, time_budget=1.0,
|
||||
)
|
||||
assert len(p._session.requests) == 1
|
||||
|
||||
def test_budget_still_allows_the_free_immediate_retry(self, monkeypatch):
|
||||
"""The 0s retry costs nothing, so a small budget can still fund it."""
|
||||
from src.providers.base import ProviderError
|
||||
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
p = TestAssetDownloadRetries()._provider([
|
||||
TestAssetDownloadRetries._Resp(500) for _ in range(9)
|
||||
])
|
||||
with pytest.raises(ProviderError):
|
||||
p._send_with_retries(
|
||||
"GET", "https://files.example/signed",
|
||||
max_retries=5, immediate_first_retry=True, time_budget=1.0,
|
||||
)
|
||||
# Attempt 1, free retry, then the 2s backoff exceeds what is left.
|
||||
assert len(p._session.requests) == 2
|
||||
|
||||
def test_read_timeout_is_clamped_to_the_remaining_budget(self, monkeypatch):
|
||||
"""So a final attempt cannot overrun the budget by a whole timeout."""
|
||||
from src.providers.base import ProviderError
|
||||
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
p = TestAssetDownloadRetries()._provider([
|
||||
TestAssetDownloadRetries._Resp(500) for _ in range(9)
|
||||
])
|
||||
with pytest.raises(ProviderError):
|
||||
p._send_with_retries(
|
||||
"GET", "https://files.example/signed",
|
||||
max_retries=5, time_budget=3.0, timeout=(10, 20),
|
||||
)
|
||||
_, _, kwargs = p._session.requests[0]
|
||||
connect_to, read_to = kwargs["timeout"]
|
||||
assert connect_to == 10
|
||||
assert read_to <= 3.0, "read timeout must not exceed the budget"
|
||||
|
||||
def test_without_a_budget_the_full_ladder_runs(self, monkeypatch):
|
||||
from src.providers.base import ProviderError
|
||||
|
||||
monkeypatch.setattr("src.providers.base.time.sleep", lambda s: None)
|
||||
p = TestAssetDownloadRetries()._provider([
|
||||
TestAssetDownloadRetries._Resp(500) for _ in range(9)
|
||||
])
|
||||
with pytest.raises(ProviderError):
|
||||
p._send_with_retries(
|
||||
"GET", "https://files.example/signed",
|
||||
max_retries=5, immediate_first_retry=True,
|
||||
)
|
||||
assert len(p._session.requests) == 6, "5 retries = 6 attempts"
|
||||
|
||||
Reference in New Issue
Block a user