Jesse.Markowitz/AIChatExporter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: retry asset downloads on 5xx and recover them across runs

Browse Source 
An asset download is two hops — mint a signed URL, then fetch the bytes —
and only the first went through _make_request. The second was a bare
session.request with no 5xx handling, which lost two images in the
2026-10-20 09:00 run: one to a 500 the retried first hop had already
recovered from, one to a 503. Both re-downloaded cleanly an hour later
(106 KB and 1.3 MB), so the bytes were there all along.

A third asset timed out and aborted its whole conversation. curl_cffi's
RequestException subclasses OSError, so the un-wrapped timeout slipped
past resolve_media's except ProviderError into the export loop's
except OSError and was reported as "File write failed for conversation
6a945155" — a conversation that had already downloaded ~20 images.
Nothing was wrong with any file write.

The retry loop is factored out of _make_request into _send_with_retries,
which returns the raw response so a bytes caller gets the same handling
as a JSON one. Hop 2 goes through it: 5xx retried, Retry-After honoured,
timeouts surfacing as ProviderError for resolve_media to placeholder.
The hop is now logged, which it never was. Assets get their own ladder —
an immediate retry, then ~2/4/8/16s with jitter kept, since a storage
incident fails many assets at once — bounded by a per-asset wall-clock
budget, because the timeout case costs 40s a go.

Beyond the retry:

- Transient failures (5xx, timeout, network) are recorded as
  pending_media so a later run re-fetches just that asset; capped at
  3 runs. 404/403 stay permanent — settled 2026-08-17 for 19 assets.
- A circuit breaker stops downloads after 5 consecutive transient
  failures, recording the rest as retryable. Permanent failures never
  trip it; any success resets the streak.
- Media failures now appear in the set ids.
  The run that lost two assets pushed a clean "OK".

Keeps FUTURE.md's per-conversation cache reset correctly closed. The
README split (FUTURE.md §2) stays otial
enough to justify breaking every anchor link.

37 new tests (414 passing). Verified live: both lost assets recovered,
doctor green, a read-only chatgpt dns.
The cross-run retry and breaker are unit-tested only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/cwa7qnGp
Claude-Session: https://claude.ai/code/session_012gCWSknsWvhLnMFwa7qnGp
This commit is contained in:
JesseMarkowitz
2026-10-20 10:32:20 -04:00
co-authored by Claude Opus 5
parent 805689c649
commit 8c3e00a520
12 changed files with 990 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+49 -1
View File
@@ -807,8 +807,19 @@ On every `joplin` run:
**This design makes every run inherently resumable.** If the tool is interrupted for any reason — rate limit, network drop, Ctrl+C, crash — simply re-run the same command. It will skip already-processed conversations and continue from where it stopped.
A conversation whose assets failed transiently also carries `pending_media` (the
asset refs still missing) and `pending_media_attempts`. While `pending_media` is
set and the attempt cap is not reached, step 2 treats the conversation as stale
so the next run re-fetches those assets — see
[Embedded images and media](#embedded-images-and-media). A run that recovers them
clears the key.
To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
To re-export a single conversation, delete its entry from `cache/manifest.json`
and re-run. (There is deliberately no `cache --reset --conversation` flag; see
`FUTURE.md`.)
---
## Troubleshooting
@@ -823,6 +834,12 @@ Note: neither token's expiry can be read client-side. Claude's `sessionKey` is a
### `429 Rate Limited`
The tool automatically pauses, saves progress, and exits with a clear message showing how many conversations were exported vs remaining. Just re-run the same export command to resume — the cache picks up exactly where it left off.
### `HTTP 500` / `503` on an asset download
The provider's asset storage, not your setup. Retries and the cross-run recovery
described in [Embedded images and media](#embedded-images-and-media) handle this;
the run reports `media failed` and the next run re-fetches. Measured 2026-10-20:
two assets lost this way to a 500 and a 503 both downloaded cleanly an hour later.
### Joplin: "JOPLIN_API_TOKEN is not set"
You need to configure the token before running the `joplin` command:
1. Open Joplin desktop
@@ -855,7 +872,38 @@ The provider's internal API may have changed. Run with `--debug`, sanitize the o
Since v0.4.0, rich content is preserved as typed blocks in the export. ChatGPT voice transcripts render as text and audio assets as `📎 File attached` placeholders with size and duration metadata. Anything the extractor doesn't recognise renders as a visible `> ⚠️ Unsupported content` block naming the type and observed keys, *and* increments a counter in the post-export summary so you can tell whether real content is being silently skipped.
### Embedded images and media
Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as `![](media/…)`. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Assets that have expired on the provider's side (older AI-generated images often do) keep their text placeholder and are tallied as `media failed` in the run summary — they're never fatal to the export. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as `![](media/…)`. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
A failed asset **never fails the export** — the conversation is archived with its
text placeholder, and the failure is tallied as `media failed` in the run summary
and in the push notification. What happens next depends on *why* it failed:
| Bucket | Meaning | Retried? |
|---|---|---|
| `expired-or-missing` | 404 — the upload was deleted server-side | No. The bytes are gone. |
| `forbidden` | 403 — metadata survives, bytes do not (see the 2026-08-17 investigation in `CHANGELOG.md`) | No. |
| `download-error` | 5xx, a timeout, a network blip | **Yes** — within the run, then on up to 3 later runs |
| `skipped-provider-unwell` | the run's circuit breaker was open | **Yes**, on a later run |
| `unparseable-ref` / `write-error` | local problem, not the provider's | No |
**Within a run**, an asset download is two requests — one for a signed URL, one
for the bytes — and each is retried on a 5xx: immediately once, then after about
2, 4, 8 and 16 seconds with jitter, honouring `Retry-After` when the server sends
one. Retries for a single asset are capped by a wall-clock budget so a hanging
endpoint cannot consume the run's whole execution window.
**Across runs**, an asset that failed transiently is recorded against its
conversation in the cache manifest, and the next run re-exports that conversation
to re-fetch it. Only the missing asset is fetched — media already on disk is
reused, and Joplin resource IDs are keyed by path, so the re-sync uploads just
the new file. After 3 unsuccessful runs the asset is treated as lost and the
conversation stops being re-exported.
**Circuit breaker.** If 5 transient failures land back to back, asset downloads
stop for the rest of the run and the remaining assets are recorded for a later
attempt. A provider's storage having a bad day should cost one quiet morning, not
a run that overruns its two-hour scheduled window. Permanent failures never count
toward this — a deleted upload says nothing about the provider's health.
### Tool output / hidden context collapsed
Since v0.6.0, content that was invisible in the ChatGPT web UI is collapsed by default to one-line placeholders like `> 🔧 Tool output — file_search (15.0 KB) — omitted`. This is ChatGPT's file-retrieval tool re-injecting your attached files on every run — it can be 90% of a conversation's bytes while containing none of the dialogue. Custom Instructions collapse the same way (`> ℹ️ Hidden context`). The post-export summary lists every collapsed message by origin with the total KB omitted. To keep everything, set `EXPORTER_HIDDEN_CONTENT=full` in `.env` or pass `--hidden-content full`.