fix: retry asset downloads on 5xx and recover them across runs

An asset download is two hops — mint a signed URL, then fetch the bytes —
and only the first went through _make_request. The second was a bare
session.request with no 5xx handling, which lost two images in the
2026-10-20 09:00 run: one to a 500 the retried first hop had already
recovered from, one to a 503. Both re-downloaded cleanly an hour later
(106 KB and 1.3 MB), so the bytes were there all along.

A third asset timed out and aborted its whole conversation. curl_cffi's
RequestException subclasses OSError, so the un-wrapped timeout slipped
past resolve_media's except ProviderError into the export loop's
except OSError and was reported as "File write failed for conversation
6a945155" — a conversation that had already downloaded ~20 images.
Nothing was wrong with any file write.

The retry loop is factored out of _make_request into _send_with_retries,
which returns the raw response so a bytes caller gets the same handling
as a JSON one. Hop 2 goes through it: 5xx retried, Retry-After honoured,
timeouts surfacing as ProviderError for resolve_media to placeholder.
The hop is now logged, which it never was. Assets get their own ladder —
an immediate retry, then ~2/4/8/16s with jitter kept, since a storage
incident fails many assets at once — bounded by a per-asset wall-clock
budget, because the timeout case costs 40s a go.

Beyond the retry:

- Transient failures (5xx, timeout, network) are recorded as
  pending_media so a later run re-fetches just that asset; capped at
  3 runs. 404/403 stay permanent — settled 2026-08-17 for 19 assets.
- A circuit breaker stops downloads after 5 consecutive transient
  failures, recording the rest as retryable. Permanent failures never
  trip it; any success resets the streak.
- Media failures now appear in the set ids.
  The run that lost two assets pushed a clean "OK".

Keeps FUTURE.md's per-conversation cache reset correctly closed. The
README split (FUTURE.md §2) stays otial
enough to justify breaking every anchor link.

37 new tests (414 passing). Verified live: both lost assets recovered,
doctor green, a read-only chatgpt dns.
The cross-run retry and breaker are unit-tested only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/cwa7qnGp
Claude-Session: https://claude.ai/code/session_012gCWSknsWvhLnMFwa7qnGp
This commit is contained in:
JesseMarkowitz
2026-10-20 10:32:20 -04:00
co-authored by Claude Opus 5
parent 805689c649
commit 8c3e00a520
12 changed files with 990 additions and 63 deletions
+28 -5
View File
@@ -51,6 +51,9 @@ from src.providers.base import (
HIDDEN_CONTENT_FULL,
HIDDEN_CONTENT_OMIT,
HIDDEN_CONTENT_PLACEHOLDER,
MEDIA_MAX_RETRIES,
MEDIA_REQUEST_TIMEOUT,
MEDIA_TIME_BUDGET,
ProviderError,
REQUEST_TIMEOUT,
VALID_HIDDEN_CONTENT_POLICIES,
@@ -764,14 +767,34 @@ class ChatGPTProvider(BaseProvider):
RuntimeError(f"No download_url in response: {meta.get('detail') or meta}"),
)
self._pace()
resp = self._session.request("GET", download_url, timeout=REQUEST_TIMEOUT)
if resp.status_code != 200:
# The signed URL is a second hop that flakes on its own account.
# Measured 2026-10-20 in one scheduled run: a 500 here *after* the
# metadata hop had recovered on its 4th try, and a 503 on a different
# asset whose metadata hop was clean first time. It therefore goes
# through the shared retry path rather than being a bare request —
# which also means a timeout arrives as ProviderError, so resolve_media
# placeholders the asset, instead of as the OSError that previously
# escaped and aborted the whole conversation.
#
# No _pace() here: _send_with_retries paces every attempt itself, and
# pacing twice would add REQUEST_DELAY to every asset in the archive.
try:
resp = self._send_with_retries(
"GET",
download_url,
max_retries=MEDIA_MAX_RETRIES,
immediate_first_retry=True,
time_budget=MEDIA_TIME_BUDGET,
timeout=MEDIA_REQUEST_TIMEOUT,
)
except ProviderError as e:
# Keep "which hop" in the message: that distinction is what made
# this diagnosable, since the two hops fail for different reasons.
raise ProviderError(
self.provider_name,
f"download_asset({file_id})",
RuntimeError(f"Signed URL returned HTTP {resp.status_code}"),
)
RuntimeError(f"Signed URL: {e.original}"),
) from e
mime = resp.headers.get("content-type") or None
file_name = meta.get("file_name") or None
logger.debug(