fix: retry asset downloads on 5xx and recover them across runs

An asset download is two hops — mint a signed URL, then fetch the bytes —
and only the first went through _make_request. The second was a bare
session.request with no 5xx handling, which lost two images in the
2026-10-20 09:00 run: one to a 500 the retried first hop had already
recovered from, one to a 503. Both re-downloaded cleanly an hour later
(106 KB and 1.3 MB), so the bytes were there all along.

A third asset timed out and aborted its whole conversation. curl_cffi's
RequestException subclasses OSError, so the un-wrapped timeout slipped
past resolve_media's except ProviderError into the export loop's
except OSError and was reported as "File write failed for conversation
6a945155" — a conversation that had already downloaded ~20 images.
Nothing was wrong with any file write.

The retry loop is factored out of _make_request into _send_with_retries,
which returns the raw response so a bytes caller gets the same handling
as a JSON one. Hop 2 goes through it: 5xx retried, Retry-After honoured,
timeouts surfacing as ProviderError for resolve_media to placeholder.
The hop is now logged, which it never was. Assets get their own ladder —
an immediate retry, then ~2/4/8/16s with jitter kept, since a storage
incident fails many assets at once — bounded by a per-asset wall-clock
budget, because the timeout case costs 40s a go.

Beyond the retry:

- Transient failures (5xx, timeout, network) are recorded as
  pending_media so a later run re-fetches just that asset; capped at
  3 runs. 404/403 stay permanent — settled 2026-08-17 for 19 assets.
- A circuit breaker stops downloads after 5 consecutive transient
  failures, recording the rest as retryable. Permanent failures never
  trip it; any success resets the streak.
- Media failures now appear in the set ids.
  The run that lost two assets pushed a clean "OK".

Keeps FUTURE.md's per-conversation cache reset correctly closed. The
README split (FUTURE.md §2) stays otial
enough to justify breaking every anchor link.

37 new tests (414 passing). Verified live: both lost assets recovered,
doctor green, a read-only chatgpt dns.
The cross-run retry and breaker are unit-tested only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/cwa7qnGp
Claude-Session: https://claude.ai/code/session_012gCWSknsWvhLnMFwa7qnGp
This commit is contained in:
JesseMarkowitz
2026-10-20 10:32:20 -04:00
co-authored by Claude Opus 5
parent 805689c649
commit 8c3e00a520
12 changed files with 990 additions and 63 deletions
+63
View File
@@ -661,3 +661,66 @@ class TestNoRealNotificationsDuringTests:
monkeypatch.setenv("NTFY_TOPIC", "some-topic")
assert "127.0.0.1" in os.getenv("NTFY_SERVER", "")
assert notify_mod.send("t", "m") is False
# ---------------------------------------------------------------------------
# Media failures in the push notification.
#
# On 2026-10-20 a scheduled run lost two assets and pushed a clean "OK" with
# no hint of it — the counts were in the log file only, which is a place you
# have to think to go and look. Counts only, never asset ids: a topic on
# public ntfy.sh is readable by anyone who knows its name.
# ---------------------------------------------------------------------------
class TestMediaFailuresInNotification:
def test_media_failures_appear_in_the_message(self):
from src.notify import format_summary
_, message, _, _ = format_summary(
{"chatgpt": {"exported": 24, "skipped": 826, "failed": 0, "media_failed": 2}},
None,
None,
)
assert "2 media failed" in message
def test_absent_when_there_are_none(self):
from src.notify import format_summary
_, message, _, _ = format_summary(
{"chatgpt": {"exported": 5, "skipped": 1, "failed": 0, "media_failed": 0}},
None,
None,
)
assert "media failed" not in message
def test_media_failures_alone_do_not_fail_the_run(self):
"""The transcript was archived and a transient asset retries later."""
from src.notify import format_summary
title, _, tags, priority = format_summary(
{"chatgpt": {"exported": 1, "skipped": 0, "failed": 0, "media_failed": 3}},
None,
None,
)
assert title.startswith("AI archive OK")
assert priority == "default"
assert tags == "white_check_mark"
def test_no_asset_ids_leak_into_the_payload(self):
from src.notify import format_summary
_, message, _, _ = format_summary(
{"chatgpt": {"exported": 1, "skipped": 0, "failed": 0, "media_failed": 1}},
None,
None,
)
assert "sediment" not in message and "file_" not in message
def test_older_summaries_without_the_key_still_render(self):
from src.notify import format_summary
_, message, _, _ = format_summary(
{"codex": {"exported": 0, "skipped": 29, "failed": 0}}, None, None
)
assert "codex: 0 exported, 29 up to date" in message