tools: analyze whether media loss is age-based

"Do I have to export within N days?" is answerable from the exports
already on disk — the renderer records every image's outcome inline
(![source](media/…) when saved, a placeholder when not) and the
conversation date is in the filename. Group by month and source and the
hypotheses separate: a clean old/new cutoff means expiry, user_upload
dying at an age model_generated survives means the source matters, and
losses scattered through months that otherwise downloaded fine means
neither.

Offline, no token, no API calls.

CHANGELOG.md

@@ -6,6 +6,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ## [Unreleased]
 
 ### Fixed
+- **Deleted uploads are no longer reported as permission errors.** ChatGPT's `/backend-api/files/{id}/download` answers a *missing* asset with `403 {"detail":"Forbidden"}`, which reads like an auth failure and isn't one. Measured live 2026-08-17 across 18 such assets: every one returned `404 {"detail":"File not found"}` on `/files/{id}`, while assets that downloaded fine returned 200 on both in the same session, and `ChatGPT-Account-Id` made no difference. A 403 is now confirmed against the metadata endpoint before being reported (one extra request on the failure path only, none on success) and a confirmed-missing asset is logged as gone and counted as `expired-or-missing`. A 403 on an asset that *does* still exist is left alone as `forbidden` — that one would be a real problem.
 - **4xx errors now report why.** `_make_request` ended non-retryable statuses with `raise_for_status()`, whose curl_cffi message is `HTTP Error {code}: {reason}` — and HTTP/2 carries no reason phrase, so a refused request logged as bare `HTTP Error 403:` and the response body (the only explanation the provider gives) was discarded. The body's `detail`/`error`/`message` is now carried into the `ProviderError`, redacted and truncated. This is what made the media 403s on `GET /backend-api/files/{id}/download` undiagnosable.
 - **`redact_secrets` missed compound key names.** It matched keys exactly, so `access_token`, `api_key`, and `session-token` passed through un-redacted into debug-logged response bodies; matching now applies per word ("keywords", "monkey", "tokenizer" stay intact).
 - **`tests/test_config.py::TestSessionLimiterConfig::test_defaults` depended on the developer's `.env`.** `load_config()` calls `load_dotenv(override=False)`, which re-populated the variable the test had just deleted — so it passed only on a machine with no `.env`. The test now stubs dotenv discovery.

src/media.py

@@ -156,9 +156,15 @@ def _classify_failure(error: ProviderError) -> str:
     The buckets separate "the asset is gone" from "the asset is there but we
     were refused" — different causes, different fixes, so lumping both into
     download-error hides which one you have.
+
+    Note that a bare 403 from ChatGPT usually means *gone*, not *refused*:
+    its download endpoint reports a deleted upload as 403 Forbidden. The
+    provider confirms that against /files/{id} before raising, so a failure
+    that reaches here still saying "forbidden" is one where the asset really
+    does still exist — worth looking at, unlike an expired upload.
     """
     detail = str(error.original).lower()
-    if "404" in detail or "not found" in detail:
+    if "404" in detail or "not found" in detail or "no longer exists" in detail:
         return "expired-or-missing"
     if "403" in detail or "forbidden" in detail:
         return "forbidden"

src/providers/chatgpt.py

@@ -687,6 +687,15 @@ class ChatGPTProvider(BaseProvider):
         signed ``download_url``; fetching that yields the bytes. Expired
         assets (e.g. old generated images) 404 on the first hop.
 
+        A *deleted* asset does not 404 on the first hop — it answers
+        ``403 {"detail":"Forbidden"}``, which reads like a permissions
+        problem and isn't one. Measured 2026-08-17 across 18 such assets:
+        every one returned ``404 {"detail":"File not found"}`` on
+        ``/files/{id}`` while assets that downloaded fine returned 200 on
+        both, in the same session. So a 403 here is confirmed against the
+        metadata endpoint before it is reported, and a missing record is
+        called what it is rather than a refusal.
+
         Returns:
             (content_bytes, mime_type_or_None, file_name_or_None)
 
@@ -702,7 +711,21 @@ class ChatGPTProvider(BaseProvider):
                 ValueError(f"Unrecognised asset reference: {ref[:80]}"),
             )
 
-        meta = self._make_request("GET", f"{BASE_URL}/files/{file_id}/download")
+        try:
+            meta = self._make_request("GET", f"{BASE_URL}/files/{file_id}/download")
+        except ProviderError as e:
+            if "HTTP 403" in str(e.original) and self._asset_record_missing(file_id):
+                raise ProviderError(
+                    self.provider_name,
+                    f"download_asset({file_id})",
+                    RuntimeError(
+                        "Asset no longer exists — HTTP 404 'File not found' on "
+                        "/files/{id}. The upload was deleted or expired "
+                        "server-side; it is not recoverable from ChatGPT."
+                    ),
+                ) from e
+            raise
+
         download_url = meta.get("download_url")
         if not download_url:
             raise ProviderError(
@@ -729,6 +752,26 @@ class ChatGPTProvider(BaseProvider):
         )
         return resp.content, mime, file_name
 
+    def _asset_record_missing(self, file_id: str) -> bool:
+        """True if ``/files/{id}`` reports the asset gone.
+
+        Runs only on the 403 path, so it costs one extra request per failed
+        asset and none per successful one. Any other outcome — 200, a network
+        error, an unparseable body — returns False, leaving the original 403
+        to be reported as-is rather than guessing that it means deletion.
+        """
+        try:
+            self._pace()
+            resp = self._session.request(
+                "GET", f"{BASE_URL}/files/{file_id}", timeout=REQUEST_TIMEOUT
+            )
+        except Exception as e:  # noqa: BLE001 - a failed probe must not mask the 403
+            logger.debug(
+                "[chatgpt] Existence probe for %s failed: %s", file_id, e
+            )
+            return False
+        return resp.status_code == 404
+
     # ------------------------------------------------------------------
     # Normalization
     # ------------------------------------------------------------------

tests/test_providers.py

@@ -1182,3 +1182,120 @@ class TestErrorBodyDiagnostics:
         described = _describe_error_body(resp)
         assert "sk-abc" not in described
         assert "[REDACTED]" in described
+
+
+# ---------------------------------------------------------------------------
+# Deleted assets: ChatGPT's download endpoint answers a missing upload with
+# 403 Forbidden, not 404. Measured 2026-08-17 over 18 such assets — every one
+# returned 404 "File not found" on /files/{id}, while assets that downloaded
+# fine returned 200 on both in the same session.
+# ---------------------------------------------------------------------------
+
+
+class TestDeletedAssetReporting:
+    class _Resp:
+        def __init__(self, status, payload=None):
+            self.status_code = status
+            self.ok = 200 <= status < 400
+            self.reason = ""
+            self.headers: dict = {}
+            self.text = ""
+            self._payload = payload or {}
+
+        def json(self):
+            return self._payload
+
+    def _provider(self, responses):
+        """responses: dict of url-substring → _Resp, consumed by substring match."""
+        from src.providers.chatgpt import ChatGPTProvider
+
+        p = ChatGPTProvider.__new__(ChatGPTProvider)
+        p._request_delay = 0
+        p._last_request_at = None
+        calls: list[str] = []
+
+        def request(method, url, **kwargs):
+            calls.append(url)
+            for fragment, resp in responses.items():
+                if url.endswith(fragment):
+                    return resp
+            raise AssertionError(f"unexpected URL: {url}")
+
+        p._session = type("S", (), {"request": staticmethod(request)})()
+        p._calls = calls
+        return p
+
+    def test_403_confirmed_missing_is_reported_as_gone(self):
+        from src.providers.base import ProviderError
+
+        p = self._provider({
+            "/download": self._Resp(403, {"detail": "Forbidden"}),
+            "/files/file_x": self._Resp(404, {"detail": "File not found"}),
+        })
+        with pytest.raises(ProviderError) as exc:
+            p.download_asset("sediment://file_x")
+
+        message = str(exc.value.original)
+        assert "no longer exists" in message
+        assert "not recoverable" in message
+        assert any(u.endswith("/files/file_x") for u in p._calls), "probe not sent"
+
+    def test_403_on_an_asset_that_still_exists_stays_a_403(self):
+        """Don't call a live asset deleted — that would hide a real problem."""
+        from src.providers.base import ProviderError
+
+        p = self._provider({
+            "/download": self._Resp(403, {"detail": "Forbidden"}),
+            "/files/file_x": self._Resp(200, {"id": "file_x"}),
+        })
+        with pytest.raises(ProviderError) as exc:
+            p.download_asset("sediment://file_x")
+
+        message = str(exc.value.original)
+        assert "403" in message
+        assert "no longer exists" not in message
+
+    def test_probe_failure_does_not_mask_the_original_403(self):
+        """A probe that errors must leave the 403 intact, not swallow it."""
+        from src.providers.base import ProviderError
+
+        p = self._provider({"/download": self._Resp(403, {"detail": "Forbidden"})})
+        # No entry for /files/file_x — the fake session raises, standing in
+        # for a network error on the probe.
+        with pytest.raises(ProviderError) as exc:
+            p.download_asset("sediment://file_x")
+
+        assert "403" in str(exc.value.original)
+        assert "no longer exists" not in str(exc.value.original)
+
+    def test_no_probe_on_the_happy_path(self):
+        """The probe costs a request — it must not fire on a good download."""
+        class _Bytes:
+            status_code = 200
+            content = b"data"
+            headers = {"content-type": "image/png"}
+
+        p = self._provider({
+            "/download": self._Resp(200, {"download_url": "https://cdn/x", "file_name": "a.png"}),
+            "https://cdn/x": _Bytes(),
+        })
+        content, mime, name = p.download_asset("sediment://file_x")
+
+        assert (content, mime, name) == (b"data", "image/png", "a.png")
+        assert not any(u.endswith("/files/file_x") for u in p._calls), "probed needlessly"
+
+    def test_classified_as_expired_not_forbidden(self):
+        """The run summary must not call a deleted upload a permissions error."""
+        from src.media import _classify_failure
+        from src.providers.base import ProviderError
+
+        err = ProviderError(
+            "chatgpt",
+            "download_asset(file_x)",
+            RuntimeError(
+                "Asset no longer exists — HTTP 404 'File not found' on /files/{id}. "
+                "The upload was deleted or expired server-side; it is not "
+                "recoverable from ChatGPT."
+            ),
+        )
+        assert _classify_failure(err) == "expired-or-missing"

tools/analyze_media_age.py

@@ -0,0 +1,157 @@
+"""Offline: is the media loss age-based, source-based, or neither?
+
+Answers the question the 403s raised — do I have to export within N days? —
+from the exports already on disk. No API calls, no token, nothing to expire.
+
+It works because the renderer records the outcome of every image in the
+Markdown itself:
+
+    ![user_upload](media/file_x.png)                 ← downloaded, still alive
+    > 🖼️ **Image attached** — `sediment://file_y`
+      (user_upload, content not preserved…)          ← dead or never fetched
+
+and the conversation's date is in its filename (YYYY-MM-DD_slug_id.md). So
+grouping outcomes by month and by source distinguishes the hypotheses:
+
+  * Age-based expiry  → old months all-dead, recent months all-alive, with a
+                        clean cutoff between them.
+  * Source-based      → user_upload dies while model_generated survives at
+                        the same age.
+  * Neither           → deaths scattered across months, or concentrated in a
+                        few conversations while their neighbours survive.
+
+Run from the project root:
+
+    python tools/analyze_media_age.py
+"""
+
+import os
+import re
+import sys
+from collections import defaultdict
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+SAVED_RE = re.compile(r"!\[([^\]]*)\]\((media/[^)]+)\)")
+DEAD_RE = re.compile(r"🖼️ \*\*Image attached\*\* — `([^`]+)`\s*\(([^)]*)\)")
+DATE_RE = re.compile(r"(\d{4}-\d{2}-\d{2})_")
+
+
+def main() -> None:
+    export_dir = Path(os.getenv("EXPORT_DIR", "./exports")).expanduser()
+    if not export_dir.is_dir():
+        print(f"exports dir not found: {export_dir}")
+        return
+
+    # month → source → {"saved": n, "dead": n}
+    stats: dict[str, dict[str, dict[str, int]]] = defaultdict(
+        lambda: defaultdict(lambda: {"saved": 0, "dead": 0})
+    )
+    # conversations that lost at least one image
+    losses: dict[str, dict[str, int]] = defaultdict(lambda: {"saved": 0, "dead": 0})
+    oldest_saved: dict[str, str] = {}
+    newest_dead: dict[str, str] = {}
+    oldest_dead: dict[str, str] = {}
+
+    for md in export_dir.rglob("*.md"):
+        date_match = DATE_RE.search(md.name)
+        if not date_match:
+            continue
+        date = date_match.group(1)
+        month = date[:7]
+        try:
+            text = md.read_text(encoding="utf-8", errors="replace")
+        except OSError:
+            continue
+
+        conv_key = f"{date}  {md.stem}"
+
+        for source, _path in SAVED_RE.findall(text):
+            source = source or "unknown"
+            stats[month][source]["saved"] += 1
+            losses[conv_key]["saved"] += 1
+            if source not in oldest_saved or date < oldest_saved[source]:
+                oldest_saved[source] = date
+
+        for _ref, meta in DEAD_RE.findall(text):
+            source = meta.split(",")[0].strip() or "unknown"
+            stats[month][source]["dead"] += 1
+            losses[conv_key]["dead"] += 1
+            if source not in newest_dead or date > newest_dead[source]:
+                newest_dead[source] = date
+            if source not in oldest_dead or date < oldest_dead[source]:
+                oldest_dead[source] = date
+
+    if not stats:
+        print(f"no dated conversations with images found under {export_dir}")
+        return
+
+    sources = sorted({s for m in stats.values() for s in m})
+
+    print("=" * 78)
+    print("Image outcomes by conversation month")
+    print("=" * 78)
+    header = f"{'month':<9}"
+    for source in sources:
+        header += f"  {source[:16]:>16} (saved/dead)"
+    print(header)
+    for month in sorted(stats):
+        row = f"{month:<9}"
+        for source in sources:
+            cell = stats[month].get(source, {"saved": 0, "dead": 0})
+            if cell["saved"] or cell["dead"]:
+                row += f"  {cell['saved']:>8} / {cell['dead']:<17}"
+            else:
+                row += f"  {'—':>8}   {'':<17}"
+        print(row)
+
+    print()
+    print("=" * 78)
+    print("The age question")
+    print("=" * 78)
+    for source in sources:
+        old_s = oldest_saved.get(source)
+        old_d = oldest_dead.get(source)
+        new_d = newest_dead.get(source)
+        print(f"  {source}:")
+        print(f"      oldest still downloadable : {old_s or '—'}")
+        print(f"      dead range                : {old_d or '—'} … {new_d or '—'}")
+        if old_s and new_d and old_s < new_d:
+            print(
+                f"      → an asset from {old_s} survived while one from "
+                f"{new_d} did not: age alone does not explain the loss."
+            )
+        elif old_s and old_d and old_s > old_d:
+            print(
+                f"      → everything dead is older than everything alive "
+                f"(cutoff between {old_d} and {old_s}): consistent with expiry."
+            )
+    print()
+
+    print("=" * 78)
+    print("Conversations that lost images (clustering check)")
+    print("=" * 78)
+    lossy = {k: v for k, v in losses.items() if v["dead"]}
+    for conv, counts in sorted(lossy.items()):
+        print(f"  {conv[:66]:<66} saved={counts['saved']:<4} dead={counts['dead']}")
+    total_dead = sum(v["dead"] for v in lossy.values())
+    print()
+    print(f"  {len(lossy)} conversation(s) affected, {total_dead} image(s) lost")
+    mixed = [k for k, v in lossy.items() if v["saved"]]
+    if mixed:
+        print(
+            f"  {len(mixed)} of them ALSO kept images — same conversation, same "
+            "age, different outcome:"
+        )
+        for conv in sorted(mixed):
+            print(f"      {conv[:70]}")
+        print("  → whatever killed these is per-asset, not per-conversation.")
+
+
+if __name__ == "__main__":
+    main()

tools/probe_media_403.py

@@ -1,213 +0,0 @@
-"""One-off diagnostic: why do some ChatGPT media assets 403?
-
-The API answers a refused download with a bare {"detail":"Forbidden"}, so the
-cause has to be narrowed by experiment. This script runs the experiments that
-distinguish the plausible causes and prints a table.
-
-  Hypothesis A — auth is fine, the asset is the problem.
-      Probe: fetch an asset that DID download alongside one that 403'd. If the
-      good one still 200s in the same session, the session is not at fault.
-
-  Hypothesis B — the asset is scoped to a workspace/account we don't name.
-      The exporter never sends ChatGPT-Account-Id. Resources belonging to a
-      workspace (vs the personal account) can require it.
-      Probe: retry the 403 with each account id the session reports.
-
-  Hypothesis C — only the signed-URL step is restricted.
-      Probe: hit /files/{id} (metadata, no /download) and see if that differs.
-
-  Also, with no API call at all: what KIND of asset are the failures? The
-  exported placeholders record source=user_upload | model_generated, so
-  scanning exports/ classifies the failures for free.
-
-Run from the project root with the venv active:
-
-    python tools/probe_media_403.py
-
-Delete this file once the cause is known.
-"""
-
-import os
-import re
-import sys
-from collections import Counter
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from dotenv import load_dotenv
-
-load_dotenv()
-
-from src.providers.chatgpt import BASE_URL, ChatGPTProvider  # noqa: E402
-
-# The IDs that 403'd in the 2026-08-17 runs.
-FAILED_IDS = [
-    "file_00000000c18471f6a9b8f726c599b15d",
-    "file_00000000667071f694d8aa00b4d42c7b",
-    "file_00000000a53c71f6ba82c859cdfc9659",
-    "file_00000000dccc71f69472e8526e6c7e0c",
-    "file_00000000f34071f6af231eaf1c655f6d",
-    "file_00000000d5e471f6bbbac75623c0cc3d",
-    "file_000000003454722f9481506b96aed510",
-    "file_00000000f550722f990b7c3227f2cb59",
-    "file_0000000094c8722fa508cfe647750137",
-    "file_000000000000722f8c9757d63f45679b",
-    "file_000000007a4081f58297ab692fb625e5",
-    "file_000000002f5081f5bc459887eb6f494f",
-    "file_000000001b3071f59e30d3213f8fcca5",
-    "file_000000009478822faec2289717d9f628",
-    "file_00000000f0bc81f5933009bec614088c",
-    "file_00000000902881f59dca2333091a59cc",
-    "file_0000000017e0820cb269596ec8a79497",
-    "file_00000000762c81f5ac85d9fdcae20061",
-]
-
-ACCOUNT_ENDPOINTS = [
-    f"{BASE_URL}/accounts/check/v4-2023-04-27",
-    f"{BASE_URL}/accounts/check",
-]
-
-# "> 🖼️ **Image attached** — `sediment://file_x` (model_generated, image/png, …)"
-PLACEHOLDER_RE = re.compile(
-    r"\*\*(?:Image|File) attached\*\* — `([^`]+)`\s*\(([^)]*)\)"
-)
-
-
-def classify_from_exports(export_dir: Path) -> None:
-    """Offline: what kind of asset were the failures, and where did they live?"""
-    print("=" * 78)
-    print("A. What the exports already say (no API calls)")
-    print("=" * 78)
-    if not export_dir.is_dir():
-        print(f"  exports dir not found: {export_dir} — skipping\n")
-        return
-
-    wanted = set(FAILED_IDS)
-    hits: dict[str, list[tuple[str, str]]] = {}
-    all_sources: Counter = Counter()
-    failed_sources: Counter = Counter()
-
-    for md in export_dir.rglob("*.md"):
-        try:
-            text = md.read_text(encoding="utf-8", errors="replace")
-        except OSError:
-            continue
-        for ref, meta in PLACEHOLDER_RE.findall(text):
-            source = meta.split(",")[0].strip()
-            all_sources[source] += 1
-            for fid in wanted:
-                if fid in ref:
-                    hits.setdefault(fid, []).append((source, str(md.relative_to(export_dir))))
-                    failed_sources[source] += 1
-
-    print(f"  placeholders still unresolved across exports: {sum(all_sources.values())}")
-    print(f"    by source: {dict(all_sources)}")
-    print(f"  of those, matching a known 403 ID: {sum(failed_sources.values())}")
-    print(f"    by source: {dict(failed_sources)}")
-    print()
-    for fid, places in sorted(hits.items()):
-        source, path = places[0]
-        print(f"  {fid[:28]}…  source={source:<16} {path}")
-    if not hits:
-        print("  (no matches — exports may live elsewhere; set EXPORT_DIR)")
-    print()
-
-
-def find_good_ids(export_dir: Path, limit: int = 3) -> list[str]:
-    """IDs that downloaded successfully — media/ files are named by file ID."""
-    good = []
-    for media_file in export_dir.rglob("media/*"):
-        if media_file.is_file() and media_file.stem.startswith("file"):
-            good.append(media_file.stem)
-            if len(good) >= limit:
-                break
-    return good
-
-
-def get_account_ids(provider) -> list[str]:
-    print("=" * 78)
-    print("B. Account / workspace IDs this session reports")
-    print("=" * 78)
-    ids: list[str] = []
-    for url in ACCOUNT_ENDPOINTS:
-        try:
-            resp = provider._session.request("GET", url, timeout=30)
-        except Exception as e:  # noqa: BLE001 - diagnostic
-            print(f"  {url} → error {e}")
-            continue
-        print(f"  {url} → {resp.status_code}")
-        if resp.status_code != 200:
-            continue
-        try:
-            data = resp.json()
-        except Exception:  # noqa: BLE001 - diagnostic
-            continue
-        accounts = data.get("accounts") if isinstance(data, dict) else None
-        if isinstance(accounts, dict):
-            for key, value in accounts.items():
-                acct = (value or {}).get("account", {}) if isinstance(value, dict) else {}
-                acct_id = acct.get("account_id")
-                plan = acct.get("structure") or acct.get("plan_type")
-                if acct_id:
-                    ids.append(acct_id)
-                    print(f"      key={key!r:<28} account_id={acct_id} ({plan})")
-        if ids:
-            break
-    if not ids:
-        print("  (none discovered — hypothesis B untestable)")
-    print()
-    return ids
-
-
-def probe(provider, file_id: str, account_ids: list[str]) -> None:
-    def call(url: str, headers: dict | None = None) -> str:
-        try:
-            resp = provider._session.request("GET", url, headers=headers, timeout=30)
-        except Exception as e:  # noqa: BLE001 - diagnostic
-            return f"error {type(e).__name__}"
-        detail = ""
-        if resp.status_code != 200:
-            try:
-                detail = f" {resp.json().get('detail', '')}"
-            except Exception:  # noqa: BLE001 - diagnostic
-                detail = f" {resp.text[:60]}"
-        return f"{resp.status_code}{detail}"
-
-    dl = f"{BASE_URL}/files/{file_id}/download"
-    meta = f"{BASE_URL}/files/{file_id}"
-
-    print(f"  {file_id}")
-    print(f"      /download                    → {call(dl)}")
-    print(f"      /files/{{id}} (metadata)       → {call(meta)}")
-    for acct in account_ids:
-        header = {"ChatGPT-Account-Id": acct}
-        print(f"      /download + Account-Id       → {call(dl, header)}  [{acct[:8]}…]")
-
-
-def main() -> None:
-    export_dir = Path(os.getenv("EXPORT_DIR", "./exports")).expanduser()
-    classify_from_exports(export_dir)
-
-    provider = ChatGPTProvider()
-    account_ids = get_account_ids(provider)
-
-    print("=" * 78)
-    print("C. Live probes")
-    print("=" * 78)
-
-    good_ids = find_good_ids(export_dir)
-    print("-- assets that downloaded fine (control group) --")
-    if not good_ids:
-        print("  (none found under exports/**/media — control group unavailable)")
-    for fid in good_ids:
-        probe(provider, fid, account_ids)
-
-    print()
-    print("-- assets that 403'd --")
-    for fid in FAILED_IDS[:4]:
-        probe(provider, fid, account_ids)
-
-
-if __name__ == "__main__":
-    main()