fix: surface the response body on 4xx so 403s are diagnosable

Media downloads logged "HTTP Error 403:" with no reason. That string is
curl_cffi's raise_for_status() format, "HTTP Error {code}: {reason}", and
HTTP/2 carries no reason phrase — so the message said nothing, and
_make_request threw the response body away. The provider's JSON `detail`
is the only explanation available for a refused asset.

- base._make_request: end non-retryable statuses with a ProviderError
  carrying the body's detail/error/message (redacted, truncated to 300
  chars) instead of a bare raise_for_status().
- media: bucket 403 as `forbidden` in the run summary, separately from
  `download-error` — "the asset is gone" and "we were refused" are
  different problems.
- utils.redact_secrets: match secret key names per word. Exact matching
  let access_token, api_key, and session-token through into logged
  bodies; "keywords"/"monkey"/"tokenizer" stay intact.
- tests/test_config.py: test_defaults depended on the absence of a local
  .env — load_config() calls load_dotenv(override=False), which restored
  the variable the test had just deleted. Stub dotenv discovery.

305 tests pass.
This commit is contained in:
JesseMarkowitz
2026-08-17 07:58:02 -04:00
parent 1f5a445ada
commit 395ea19ca8
8 changed files with 193 additions and 8 deletions
+19 -3
View File
@@ -76,11 +76,27 @@ def build_export_path(
return base_dir.joinpath(*parts) / filename
def _is_sensitive_key(key: object) -> bool:
"""True if a mapping key names a secret.
Matches the whole key and each of its underscore/dash-separated words, so
compound names carry too: exact-match alone let ``access_token`` and
``api_key`` through into logged response bodies. Word-level matching keeps
innocent keys ("keywords", "monkey") intact.
"""
if not isinstance(key, str):
return False
lowered = key.lower()
if lowered in _SENSITIVE_KEYS:
return True
return any(part in _SENSITIVE_KEYS for part in re.split(r"[^a-z0-9]+", lowered))
def redact_secrets(data: object) -> object:
"""Recursively redact sensitive values from a dict/list for safe logging.
Keys matching _SENSITIVE_KEYS (case-insensitive) have their values
replaced with "[REDACTED]".
Keys naming a secret (see _is_sensitive_key) have their values replaced
with "[REDACTED]".
Args:
data: Any JSON-serializable object.
@@ -90,7 +106,7 @@ def redact_secrets(data: object) -> object:
"""
if isinstance(data, dict):
return {
k: "[REDACTED]" if k.lower() in _SENSITIVE_KEYS else redact_secrets(v)
k: "[REDACTED]" if _is_sensitive_key(k) else redact_secrets(v)
for k, v in data.items()
}
if isinstance(data, list):