fix: retry asset downloads on 5xx and recover them across runs
An asset download is two hops — mint a signed URL, then fetch the bytes — and only the first went through _make_request. The second was a bare session.request with no 5xx handling, which lost two images in the 2026-10-20 09:00 run: one to a 500 the retried first hop had already recovered from, one to a 503. Both re-downloaded cleanly an hour later (106 KB and 1.3 MB), so the bytes were there all along. A third asset timed out and aborted its whole conversation. curl_cffi's RequestException subclasses OSError, so the un-wrapped timeout slipped past resolve_media's except ProviderError into the export loop's except OSError and was reported as "File write failed for conversation 6a945155" — a conversation that had already downloaded ~20 images. Nothing was wrong with any file write. The retry loop is factored out of _make_request into _send_with_retries, which returns the raw response so a bytes caller gets the same handling as a JSON one. Hop 2 goes through it: 5xx retried, Retry-After honoured, timeouts surfacing as ProviderError for resolve_media to placeholder. The hop is now logged, which it never was. Assets get their own ladder — an immediate retry, then ~2/4/8/16s with jitter kept, since a storage incident fails many assets at once — bounded by a per-asset wall-clock budget, because the timeout case costs 40s a go. Beyond the retry: - Transient failures (5xx, timeout, network) are recorded as pending_media so a later run re-fetches just that asset; capped at 3 runs. 404/403 stay permanent — settled 2026-08-17 for 19 assets. - A circuit breaker stops downloads after 5 consecutive transient failures, recording the rest as retryable. Permanent failures never trip it; any success resets the streak. - Media failures now appear in the set ids. The run that lost two assets pushed a clean "OK". Keeps FUTURE.md's per-conversation cache reset correctly closed. The README split (FUTURE.md §2) stays otial enough to justify breaking every anchor link. 37 new tests (414 passing). Verified live: both lost assets recovered, doctor green, a read-only chatgpt dns. The cross-run retry and breaker are unit-tested only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/cwa7qnGp Claude-Session: https://claude.ai/code/session_012gCWSknsWvhLnMFwa7qnGp
This commit is contained in:
co-authored by
Claude Opus 5
parent
805689c649
commit
8c3e00a520
@@ -661,3 +661,66 @@ class TestNoRealNotificationsDuringTests:
|
||||
monkeypatch.setenv("NTFY_TOPIC", "some-topic")
|
||||
assert "127.0.0.1" in os.getenv("NTFY_SERVER", "")
|
||||
assert notify_mod.send("t", "m") is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Media failures in the push notification.
|
||||
#
|
||||
# On 2026-10-20 a scheduled run lost two assets and pushed a clean "OK" with
|
||||
# no hint of it — the counts were in the log file only, which is a place you
|
||||
# have to think to go and look. Counts only, never asset ids: a topic on
|
||||
# public ntfy.sh is readable by anyone who knows its name.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestMediaFailuresInNotification:
|
||||
def test_media_failures_appear_in_the_message(self):
|
||||
from src.notify import format_summary
|
||||
|
||||
_, message, _, _ = format_summary(
|
||||
{"chatgpt": {"exported": 24, "skipped": 826, "failed": 0, "media_failed": 2}},
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert "2 media failed" in message
|
||||
|
||||
def test_absent_when_there_are_none(self):
|
||||
from src.notify import format_summary
|
||||
|
||||
_, message, _, _ = format_summary(
|
||||
{"chatgpt": {"exported": 5, "skipped": 1, "failed": 0, "media_failed": 0}},
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert "media failed" not in message
|
||||
|
||||
def test_media_failures_alone_do_not_fail_the_run(self):
|
||||
"""The transcript was archived and a transient asset retries later."""
|
||||
from src.notify import format_summary
|
||||
|
||||
title, _, tags, priority = format_summary(
|
||||
{"chatgpt": {"exported": 1, "skipped": 0, "failed": 0, "media_failed": 3}},
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert title.startswith("AI archive OK")
|
||||
assert priority == "default"
|
||||
assert tags == "white_check_mark"
|
||||
|
||||
def test_no_asset_ids_leak_into_the_payload(self):
|
||||
from src.notify import format_summary
|
||||
|
||||
_, message, _, _ = format_summary(
|
||||
{"chatgpt": {"exported": 1, "skipped": 0, "failed": 0, "media_failed": 1}},
|
||||
None,
|
||||
None,
|
||||
)
|
||||
assert "sediment" not in message and "file_" not in message
|
||||
|
||||
def test_older_summaries_without_the_key_still_render(self):
|
||||
from src.notify import format_summary
|
||||
|
||||
_, message, _, _ = format_summary(
|
||||
{"codex": {"exported": 0, "skipped": 29, "failed": 0}}, None, None
|
||||
)
|
||||
assert "codex: 0 exported, 29 up to date" in message
|
||||
|
||||
Reference in New Issue
Block a user