fix: retry asset downloads on 5xx and recover them across runs
An asset download is two hops — mint a signed URL, then fetch the bytes — and only the first went through _make_request. The second was a bare session.request with no 5xx handling, which lost two images in the 2026-10-20 09:00 run: one to a 500 the retried first hop had already recovered from, one to a 503. Both re-downloaded cleanly an hour later (106 KB and 1.3 MB), so the bytes were there all along. A third asset timed out and aborted its whole conversation. curl_cffi's RequestException subclasses OSError, so the un-wrapped timeout slipped past resolve_media's except ProviderError into the export loop's except OSError and was reported as "File write failed for conversation 6a945155" — a conversation that had already downloaded ~20 images. Nothing was wrong with any file write. The retry loop is factored out of _make_request into _send_with_retries, which returns the raw response so a bytes caller gets the same handling as a JSON one. Hop 2 goes through it: 5xx retried, Retry-After honoured, timeouts surfacing as ProviderError for resolve_media to placeholder. The hop is now logged, which it never was. Assets get their own ladder — an immediate retry, then ~2/4/8/16s with jitter kept, since a storage incident fails many assets at once — bounded by a per-asset wall-clock budget, because the timeout case costs 40s a go. Beyond the retry: - Transient failures (5xx, timeout, network) are recorded as pending_media so a later run re-fetches just that asset; capped at 3 runs. 404/403 stay permanent — settled 2026-08-17 for 19 assets. - A circuit breaker stops downloads after 5 consecutive transient failures, recording the rest as retryable. Permanent failures never trip it; any success resets the streak. - Media failures now appear in the set ids. The run that lost two assets pushed a clean "OK". Keeps FUTURE.md's per-conversation cache reset correctly closed. The README split (FUTURE.md §2) stays otial enough to justify breaking every anchor link. 37 new tests (414 passing). Verified live: both lost assets recovered, doctor green, a read-only chatgpt dns. The cross-run retry and breaker are unit-tested only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/cwa7qnGp Claude-Session: https://claude.ai/code/session_012gCWSknsWvhLnMFwa7qnGp
This commit is contained in:
co-authored by
Claude Opus 5
parent
805689c649
commit
8c3e00a520
@@ -51,6 +51,9 @@ from src.providers.base import (
|
||||
HIDDEN_CONTENT_FULL,
|
||||
HIDDEN_CONTENT_OMIT,
|
||||
HIDDEN_CONTENT_PLACEHOLDER,
|
||||
MEDIA_MAX_RETRIES,
|
||||
MEDIA_REQUEST_TIMEOUT,
|
||||
MEDIA_TIME_BUDGET,
|
||||
ProviderError,
|
||||
REQUEST_TIMEOUT,
|
||||
VALID_HIDDEN_CONTENT_POLICIES,
|
||||
@@ -764,14 +767,34 @@ class ChatGPTProvider(BaseProvider):
|
||||
RuntimeError(f"No download_url in response: {meta.get('detail') or meta}"),
|
||||
)
|
||||
|
||||
self._pace()
|
||||
resp = self._session.request("GET", download_url, timeout=REQUEST_TIMEOUT)
|
||||
if resp.status_code != 200:
|
||||
# The signed URL is a second hop that flakes on its own account.
|
||||
# Measured 2026-10-20 in one scheduled run: a 500 here *after* the
|
||||
# metadata hop had recovered on its 4th try, and a 503 on a different
|
||||
# asset whose metadata hop was clean first time. It therefore goes
|
||||
# through the shared retry path rather than being a bare request —
|
||||
# which also means a timeout arrives as ProviderError, so resolve_media
|
||||
# placeholders the asset, instead of as the OSError that previously
|
||||
# escaped and aborted the whole conversation.
|
||||
#
|
||||
# No _pace() here: _send_with_retries paces every attempt itself, and
|
||||
# pacing twice would add REQUEST_DELAY to every asset in the archive.
|
||||
try:
|
||||
resp = self._send_with_retries(
|
||||
"GET",
|
||||
download_url,
|
||||
max_retries=MEDIA_MAX_RETRIES,
|
||||
immediate_first_retry=True,
|
||||
time_budget=MEDIA_TIME_BUDGET,
|
||||
timeout=MEDIA_REQUEST_TIMEOUT,
|
||||
)
|
||||
except ProviderError as e:
|
||||
# Keep "which hop" in the message: that distinction is what made
|
||||
# this diagnosable, since the two hops fail for different reasons.
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"download_asset({file_id})",
|
||||
RuntimeError(f"Signed URL returned HTTP {resp.status_code}"),
|
||||
)
|
||||
RuntimeError(f"Signed URL: {e.original}"),
|
||||
) from e
|
||||
mime = resp.headers.get("content-type") or None
|
||||
file_name = meta.get("file_name") or None
|
||||
logger.debug(
|
||||
|
||||
Reference in New Issue
Block a user