Compare commits
3
Commits
d3745e1de4
...
23c6e1f512
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
23c6e1f512 | ||
|
|
64068bb19b | ||
|
|
b6ce636891 |
+10
-2
@@ -6,10 +6,18 @@
|
||||
|
||||
# --- ChatGPT ---
|
||||
# How to get: open chatgpt.com in Chrome → F12 → Application tab
|
||||
# → Cookies → https://chatgpt.com → find the two cookie chunks:
|
||||
# → Cookies → https://chatgpt.com → find the session token cookie. Chrome splits a
|
||||
# cookie only above ~4KB, so you will see ONE of these two layouts:
|
||||
#
|
||||
# __Secure-next-auth.session-token (the whole value) → CHATGPT_SESSION_TOKEN
|
||||
# (leave _1 empty)
|
||||
# or, when the token was large enough to be split:
|
||||
# __Secure-next-auth.session-token.0 (starts with "eyJ") → CHATGPT_SESSION_TOKEN
|
||||
# __Secure-next-auth.session-token.1 (the remainder) → CHATGPT_SESSION_TOKEN_1
|
||||
# Token type: JWE. Typically valid for ~7 days.
|
||||
#
|
||||
# CHATGPT_SESSION_TOKEN_1 is OPTIONAL — leave it empty when there is no .1 cookie.
|
||||
# But if a .1 cookie does exist, you must copy it: a partial .0 fails silently
|
||||
# (HTTP 200 with no accessToken). Token type: JWE. Typically valid for ~7 days.
|
||||
CHATGPT_SESSION_TOKEN=
|
||||
CHATGPT_SESSION_TOKEN_1=
|
||||
|
||||
|
||||
@@ -6,6 +6,32 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
- **Every scheduled Claude Code sync since 2026-09-24 died with `RecursionError`.** Claude Code's `fork` subagents write a transcript that opens with a copy of the parent turn that spawned them — the fork's own `Agent` call included. `_extract_messages` folds a subagent inline whenever it meets its spawn call, so it met that copy inside the fork, folded the same fork again, and recursed until Python's limit. One fork anywhere in `~/.claude/projects` was enough to fail the whole provider; seven existed across three sessions, and the Codex half of the run, unaffected, hid the cause behind a generic exit 1.
|
||||
|
||||
The recursive pass now carries the spawn ids being expanded around it and skips a tool_use whose id is among them — the enclosing subagent block already stands for that call. Because the set accumulates, a longer cycle (A spawns B, whose transcript re-spawns A) stops too. The fork's `<fork-boilerplate>` preamble — generic worker rules the harness prepends — is now stripped with the other harness tags, so a fork block opens with its actual directive.
|
||||
|
||||
`TestSubagentFold.test_fork_containing_its_own_spawn_call` reproduces the on-disk shape (a `fork-context-ref` record, the copied spawn call, the boilerplate-wrapped directive) and fails with the original `RecursionError` against the unfixed code. Verified against the real archive: all 85 sessions normalize, the seven forks each render as one subagent block.
|
||||
|
||||
- **A scheduled run that crashed sent no notification, and the providers that survived said "OK".** `sync` pushes its ntfy result from the end of a run it finished, so the `RecursionError` above — and any crash, any exit before the sync starts (the ToS gate, a cache error), a launcher that cannot build its venv — sent nothing at all. Worse, the unit runs each provider as its own `sync`, so codex kept pushing a low-priority "AI archive OK" every morning for the eleven days claude-code was dead: a broken provider was indistinguishable from a quiet day.
|
||||
|
||||
The systemd unit's `ExecStart` is now `scheduling/run-sync.sh`, which carries the old per-provider loop and pushes a high-priority **FAILED** notification for any run that exits non-zero without the app's "Sync completed with failures" banner (printed right after the app's own push, so an app-reported failure is not reported twice). The push names the provider and, for a crash, the exception class only — `claude-code: crashed (RecursionError)` — never its message, holding to `src/notify.py`'s counts-only rule for a topic anyone can read. It honours `NTFY_NOTIFY=off` and reads `NTFY_*` the way the app does: environment first, then `.env`. Re-run `install-systemd-timer.sh` to pick it up. The Windows task has no equivalent yet.
|
||||
|
||||
Verified against a fake launcher and a local capture server: a crash pushes with the class and without the message, an exit before the sync pushes, an app-reported failure and a success push nothing extra, `off` pushes nothing, and the run still exits 1 if any provider failed.
|
||||
|
||||
- **An expired Claude session key reported a raw JSON dump instead of how to fix it.** `_make_request` routed only **401** to the auth handler (`src/providers/base.py`), and claude.ai does not use 401 — an invalid or expired `sessionKey` comes back as `403 permission_error` with `details.error_code = account_session_invalid`. So the one message that names the cookie, its ~30-day lifetime and the DevTools path to refresh it could never fire for Claude. What the user got instead was the generic 4xx path: `HTTP 403 — error: {'type': 'permission_error', 'message': 'Invalid authorization'…}`, which reads like a permissions problem with the account and not like "your key expired, here is how to replace it."
|
||||
|
||||
Measured live 2026-09-20 against `GET /api/organizations`: a valid key returns 200, while an expired key, a deliberately malformed key and **no cookie at all** return byte-identical 403s carrying that code — i.e. the API treats a dead session as an absent one. This is the same mistake as the ChatGPT media 403s below: assuming 403 means "forbidden" when the service uses it for "unauthenticated."
|
||||
|
||||
Auth detection is now a provider decision rather than a hardcoded status. `BaseProvider._is_auth_failure(response)` defaults to 401 and `ClaudeProvider` overrides it to add 403 **matched on `account_session_invalid`**, not on the bare status — so a genuine permission error, which carries a different code, is still reported as itself rather than being mislabelled an expired key. `_handle_401` is renamed `_handle_auth_failure` and takes the response, because a handler named for one status that must handle two is how this stayed hidden; its messages now state the status actually observed instead of asserting "401 Unauthorized". ChatGPT is unaffected: it does not override the default, so the deleted-asset 403 path is untouched.
|
||||
|
||||
Seven regression tests cover the split (`TestAuthFailureDetection`), including the two that matter most: a Claude 403 with a *different* error code must **not** be treated as an auth failure, and a ChatGPT 403 must not either. The docs that repeated the wrong premise — `README.md`'s expiry table and "When Tokens Expire" section, the `auth` wizard's on-screen note, and the `ClaudeProvider` docstring — are corrected in the same change.
|
||||
|
||||
- **The docs claimed both ChatGPT cookie chunks were required; they are not.** `README.md` stated flatly that "ChatGPT splits large session tokens across two cookies to stay under the browser's 4KB cookie limit. Both are required," and `.env.example` documented only the chunked layout — so a machine whose session token happens to fit in a single `__Secure-next-auth.session-token` cookie looked broken, with the user hunting for a `.1` that does not exist. Chrome splits a cookie only above ~4KB, so the layout varies by session size and the *same account* can be chunked on one machine and not on another.
|
||||
|
||||
The code was already correct: `CHATGPT_SESSION_TOKEN_1` is optional (`src/providers/chatgpt.py:162`) and the `auth` wizard already told you to paste a lone cookie into `.0` and leave `.1` blank. Only the reference docs were wrong, and they are the ones read when setting up a new machine.
|
||||
|
||||
Measured 2026-09-20 against `/api/auth/session`, reassembling a real 4089-byte token to test each naming: chunked `.0`+`.1` → 200 with an `accessToken`; the whole value under the unchunked name → 200 with an `accessToken`; the whole value under `.0` alone → 200 with an `accessToken`. The server reassembles a complete value sent under `.0`, so both layouts authenticate as the code already assumed. A *partial* `.0` with its `.1` omitted is the one combination that fails, and it fails **silently** — HTTP 200 with no `accessToken` rather than an error — which is now documented in both files alongside the correction.
|
||||
|
||||
- **The test suite sent real push notifications to the developer's phone.** `TestSyncCommand` invokes the actual `sync` command, which calls `load_config()`, which calls `load_dotenv()` — so the real `.env` was loaded and its live `NTFY_TOPIC` used for the POST. Every `pytest` run fired three or four pushes, including a fabricated "codex: 3 conversation(s) failed to export" straight out of a fixture, which is worse than noise: it reports a failure that never happened. Nothing appeared in `cache/logs/exporter.log` to explain it, because every test invocation passes `--no-log-file`.
|
||||
|
||||
A `tests/conftest.py` autouse fixture now neutralises the environment for every test: `NTFY_TOPIC`/`NTFY_TOKEN` are emptied, and `NTFY_SERVER` and `JOPLIN_API_URL` are pointed at a closed local port, so a stray topic cannot reach the internet and a test cannot write notes into a real Joplin instance. The values are **emptied rather than deleted** — `load_dotenv(override=False)` skips only keys already present, so deleting one lets `.env` put it back. Verified by instrumenting `requests` across a full run: zero outbound requests, where the same instrumentation without the fixture records POSTs to the live ntfy topic.
|
||||
|
||||
@@ -169,8 +169,8 @@ The wizard detects your OS, shows the correct DevTools shortcut, and writes the
|
||||
|
||||
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
||||
|----------|-------------|----------|-----------------|
|
||||
| ChatGPT | `__Secure-next-auth.session-token.0` + `.1` | refresh ~weekly | `error` field of `/api/auth/session` — `doctor` reports "ChatGPT token active". The token is an encrypted JWE, so its `exp` is **not** readable client-side, and the `expires` field is a misleading rolling window; the `error` (`RefreshAccessTokenError` when dead) is the honest signal. |
|
||||
| Claude | `sessionKey` | ~30 days | Opaque token — only detectable via 401 response |
|
||||
| ChatGPT | `__Secure-next-auth.session-token` (split into `.0` + `.1` when over ~4KB) | refresh ~weekly | `error` field of `/api/auth/session` — `doctor` reports "ChatGPT token active". The token is an encrypted JWE, so its `exp` is **not** readable client-side, and the `expires` field is a misleading rolling window; the `error` (`RefreshAccessTokenError` when dead) is the honest signal. |
|
||||
| Claude | `sessionKey` | ~30 days | Opaque token — only detectable from an API rejection. claude.ai answers an invalid session with **403** `permission_error` / `account_session_invalid`, **not** 401; `doctor` reports it on the "Claude API reachable" row. |
|
||||
|
||||
### Finding Tokens in Chrome DevTools
|
||||
|
||||
@@ -180,20 +180,38 @@ The wizard detects your OS, shows the correct DevTools shortcut, and writes the
|
||||
4. In the left panel, expand **Cookies** and click the site URL
|
||||
5. Find the cookie by name and copy its **Value**
|
||||
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find **two** cookies:
|
||||
- `__Secure-next-auth.session-token.0` — copy Value (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`
|
||||
- `__Secure-next-auth.session-token.1` — copy Value → `CHATGPT_SESSION_TOKEN_1`
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find the session token cookie. You will
|
||||
see **one of two layouts**, depending on how large your session token is:
|
||||
|
||||
ChatGPT splits large session tokens across two cookies to stay under the browser's 4KB cookie limit. Both are required.
|
||||
- **One cookie**, `__Secure-next-auth.session-token` — copy Value → `CHATGPT_SESSION_TOKEN`, and leave `CHATGPT_SESSION_TOKEN_1` empty.
|
||||
- **Two cookies**, `__Secure-next-auth.session-token.0` and `.1` — copy `.0` (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`, and `.1` → `CHATGPT_SESSION_TOKEN_1`.
|
||||
|
||||
Chrome splits a cookie only when it exceeds ~4KB, so a larger session is chunked
|
||||
and a smaller one is not — the same account can differ from machine to machine.
|
||||
`CHATGPT_SESSION_TOKEN_1` is optional; both layouts authenticate, because the
|
||||
server reassembles a complete value sent under the `.0` name.
|
||||
|
||||
What does *not* work is sending a **partial** chunk — `.0` on its own when a `.1`
|
||||
exists. That fails silently: `/api/auth/session` answers HTTP 200 with no
|
||||
`accessToken` rather than an error. If you see two cookies, copy both.
|
||||
|
||||
**Claude:** go to `https://claude.ai` → find `sessionKey` → copy Value
|
||||
|
||||
### When Tokens Expire
|
||||
|
||||
When a token expires you'll see a `401 Unauthorized` error. To refresh:
|
||||
An expired token shows up as an authentication error naming the cookie to
|
||||
refresh and how. The status differs by provider — ChatGPT reports 401, while
|
||||
claude.ai reports **403 "Invalid authorization"** (`account_session_invalid`) —
|
||||
so don't read a 403 from Claude as a permissions problem with your account.
|
||||
|
||||
To refresh:
|
||||
- Re-run the `auth` wizard: `ai-chat-exporter auth`
|
||||
- Or manually update the value in your `.env` file
|
||||
|
||||
`ai-chat-exporter doctor` is the quickest check: the "token set" rows only test
|
||||
that a value is present, so an expired credential passes those and fails on the
|
||||
"API reachable" row.
|
||||
|
||||
---
|
||||
|
||||
## The `auth` Command
|
||||
@@ -411,7 +429,8 @@ cache on the next run that finds Joplin up.
|
||||
providers in a single action rather than one action each, because systemd
|
||||
`oneshot` stops at the first failing `ExecStart` and Task Scheduler reports only
|
||||
the last action's result. Every provider is attempted; the run still exits
|
||||
non-zero if any failed.
|
||||
non-zero if any failed. On Linux the loop is `scheduling/run-sync.sh`, which the
|
||||
unit's `ExecStart` calls.
|
||||
|
||||
### Getting notified
|
||||
|
||||
@@ -439,6 +458,16 @@ high priority with an alert tag, so a failed archive is distinguishable from a
|
||||
quiet one on your phone. `NTFY_NOTIFY=failure` notifies only on failure; `off`
|
||||
disables it; `--notify` / `--no-notify` override per run.
|
||||
|
||||
`sync` can only push from the end of a run it finished. A crash, an exit before
|
||||
the sync starts (the ToS gate, a cache error) or a launcher that can't build its
|
||||
venv sends nothing — and since each provider pushes separately, the ones that
|
||||
succeeded still say "OK", so a dead provider looks like a quiet day. On Linux,
|
||||
`scheduling/run-sync.sh` closes that gap: any run that exits non-zero without the
|
||||
app having reported it gets a high-priority **FAILED** push naming the provider
|
||||
and, for a crash, the exception's class (`claude-code: crashed (RecursionError)`)
|
||||
— the class only, never its message, which can carry a conversation title. The
|
||||
traceback is in the journal. The Windows task has no such backstop yet.
|
||||
|
||||
The message includes the **machine name**, which matters because both machines
|
||||
archive into one topic. It contains counts only — never conversation titles. A
|
||||
topic on public ntfy.sh is readable by anyone who knows its name, so if you want
|
||||
@@ -776,12 +805,12 @@ To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `401 Unauthorized`
|
||||
### `Authentication failed` (401, or 403 from Claude)
|
||||
Your session token has expired.
|
||||
- Run `ai-chat-exporter auth` to get a new token interactively
|
||||
- Or manually copy a fresh cookie value into your `.env` file
|
||||
|
||||
Note: Claude's `sessionKey` is an opaque string — the only way to know it's expired is the 401 error. ChatGPT JWTs have an `exp` claim that the `doctor` command can decode and display.
|
||||
Note: neither token's expiry can be read client-side. Claude's `sessionKey` is an opaque string, and claude.ai reports an invalid one as **403** "Invalid authorization" (`account_session_invalid`), not 401. ChatGPT's token is an encrypted JWE; `doctor` reads the `error` field of `/api/auth/session` instead. See [When Tokens Expire](#when-tokens-expire).
|
||||
|
||||
### `429 Rate Limited`
|
||||
The tool automatically pauses, saves progress, and exits with a clear message showing how many conversations were exported vs remaining. Just re-run the same export command to resume — the cache picks up exactly where it left off.
|
||||
|
||||
@@ -44,10 +44,12 @@ fi
|
||||
|
||||
[ ${#PROVIDERS[@]} -eq 0 ] && PROVIDERS=("all")
|
||||
|
||||
if [ ! -x "$REPO/ai-chat-exporter" ]; then
|
||||
echo "error: $REPO/ai-chat-exporter is missing or not executable." >&2
|
||||
for f in "$REPO/ai-chat-exporter" "$REPO/scheduling/run-sync.sh"; do
|
||||
if [ ! -x "$f" ]; then
|
||||
echo "error: $f is missing or not executable." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
mkdir -p "$UNIT_DIR"
|
||||
|
||||
@@ -65,12 +67,10 @@ mkdir -p "$UNIT_DIR"
|
||||
echo "Type=oneshot"
|
||||
echo "WorkingDirectory=$REPO"
|
||||
echo "Environment=AI_CHAT_EXPORTER_QUIET_CWD=1"
|
||||
# One ExecStart per provider would stop at the first failure, silently
|
||||
# skipping the rest — an expired ChatGPT token would mean codex never runs.
|
||||
# Loop instead, so every provider is attempted and the unit still reports
|
||||
# failure if any of them failed.
|
||||
printf 'ExecStart=/bin/sh -c '\''rc=0; for p in %s; do "$0" sync --provider "$p" --joplin-optional || rc=1; done; exit $rc'\'' %s\n' \
|
||||
"${PROVIDERS[*]}" "$REPO/ai-chat-exporter"
|
||||
echo "Environment=AICHAT_SYNC_UNIT=$NAME"
|
||||
# run-sync.sh attempts every provider even after one fails, and pushes a
|
||||
# FAILED notification for any run that died without sending its own.
|
||||
echo "ExecStart=$REPO/scheduling/run-sync.sh ${PROVIDERS[*]}"
|
||||
} > "$UNIT_DIR/$NAME.service"
|
||||
|
||||
# Persistent=true runs a missed schedule at the next boot — the machine being
|
||||
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run `ai-chat-exporter sync` once per provider — the ExecStart of the systemd
|
||||
# unit that install-systemd-timer.sh writes.
|
||||
#
|
||||
# ./scheduling/run-sync.sh claude-code codex
|
||||
#
|
||||
# Every provider is attempted even after one fails, and the exit code is
|
||||
# non-zero if any failed — one ExecStart per provider would stop at the first.
|
||||
#
|
||||
# The app pushes its own ntfy result, but only from the end of a run it
|
||||
# finished. A crash, a non-zero exit before the sync starts (the terms-of-service
|
||||
# gate, a cache error) or a launcher that can't build its venv sends nothing, and
|
||||
# because each provider pushes separately, the providers that did succeed still
|
||||
# send "OK" — so a broken one looks like a quiet day. This script pushes a FAILED
|
||||
# notification for any run that exited non-zero without the app having reported
|
||||
# it. (Its "Sync completed with failures" banner prints right after its push.)
|
||||
#
|
||||
# The push carries the provider, the exit code and, for a crash, the exception's
|
||||
# class name — never its message. Same counts-only rule as src/notify.py: on a
|
||||
# public ntfy topic anyone who guesses the name can read it, and exception text
|
||||
# can carry conversation titles. The full traceback is in the journal.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
REPO="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
LAUNCHER="$REPO/ai-chat-exporter"
|
||||
|
||||
# NTFY_* as the app resolves them: the environment wins, then .env.
|
||||
env_value() {
|
||||
local name=$1 value=${!1:-}
|
||||
if [ -z "$value" ] && [ -f "$REPO/.env" ]; then
|
||||
value=$(sed -n "s/^[[:space:]]*$name[[:space:]]*=[[:space:]]*//p" "$REPO/.env" | tail -n 1)
|
||||
value=${value%%[[:space:]]#*}
|
||||
value=${value%"${value##*[![:space:]]}"}
|
||||
value=${value#[\"\']}
|
||||
value=${value%[\"\']}
|
||||
fi
|
||||
printf '%s' "$value"
|
||||
}
|
||||
|
||||
push_failure() {
|
||||
local body=$1 topic server token policy
|
||||
topic=$(env_value NTFY_TOPIC)
|
||||
policy=$(env_value NTFY_NOTIFY | tr '[:upper:]' '[:lower:]')
|
||||
if [ -z "$topic" ] || [ "$policy" = "off" ]; then
|
||||
return 0
|
||||
fi
|
||||
server=$(env_value NTFY_SERVER)
|
||||
server=${server:-https://ntfy.sh}
|
||||
token=$(env_value NTFY_TOKEN)
|
||||
|
||||
local args=(-fsS --max-time 15 -o /dev/null
|
||||
-H "Title: AI archive FAILED - $(hostname -s)"
|
||||
-H "Tags: rotating_light" -H "Priority: high"
|
||||
--data-binary "$body")
|
||||
[ -n "$token" ] && args+=(-H "Authorization: Bearer $token")
|
||||
curl "${args[@]}" "${server%/}/$topic" \
|
||||
|| echo "run-sync: could not send the failure notification" >&2
|
||||
}
|
||||
|
||||
[ $# -eq 0 ] && set -- all
|
||||
|
||||
rc=0
|
||||
for provider in "$@"; do
|
||||
out=$(mktemp)
|
||||
"$LAUNCHER" sync --provider "$provider" --joplin-optional 2>&1 | tee "$out"
|
||||
status=${PIPESTATUS[0]}
|
||||
if [ "$status" -ne 0 ]; then
|
||||
rc=1
|
||||
if ! grep -q "Sync completed with failures" "$out"; then
|
||||
crash=$(grep -oE '^[A-Za-z_][A-Za-z0-9_.]*(Error|Exception)\b' "$out" | tail -n 1)
|
||||
if [ -n "$crash" ]; then
|
||||
reason="crashed ($crash)"
|
||||
else
|
||||
reason="exited $status before reporting a result"
|
||||
fi
|
||||
push_failure "$provider: $reason
|
||||
journalctl --user -u ${AICHAT_SYNC_UNIT:-aichat-sync} -n 100"
|
||||
fi
|
||||
fi
|
||||
rm -f "$out"
|
||||
done
|
||||
exit "$rc"
|
||||
+4
-1
@@ -258,7 +258,10 @@ def _auth_claude(os_name: str) -> None:
|
||||
console.print("2. Press [bold]F12[/bold] to open DevTools → Application tab.")
|
||||
console.print("3. Expand [bold]Cookies[/bold] → [bold]https://claude.ai[/bold]")
|
||||
console.print("4. Find [bold]sessionKey[/bold] → copy the Value.")
|
||||
console.print(" (Note: Claude tokens expire after ~30 days; a 401 error is the only signal.)")
|
||||
console.print(
|
||||
" (Note: Claude tokens expire after ~30 days; an API rejection — 403 "
|
||||
"'Invalid authorization' — is the only signal.)"
|
||||
)
|
||||
console.print("5. Paste it below (input is hidden).\n")
|
||||
|
||||
key = click.prompt("Claude session key", hide_input=True, default="", show_default=False).strip()
|
||||
|
||||
+29
-10
@@ -335,7 +335,8 @@ class BaseProvider(ABC):
|
||||
Parsed JSON response body.
|
||||
|
||||
Raises:
|
||||
ProviderError: On 401, exhausted retries, or unrecoverable errors.
|
||||
ProviderError: On an authentication failure, exhausted retries, or
|
||||
unrecoverable errors.
|
||||
"""
|
||||
kwargs.setdefault("timeout", REQUEST_TIMEOUT)
|
||||
|
||||
@@ -361,14 +362,19 @@ class BaseProvider(ABC):
|
||||
elapsed_ms,
|
||||
)
|
||||
|
||||
# ── 401: token expired / invalid ──────────────────────────
|
||||
if response.status_code == 401:
|
||||
self._handle_401()
|
||||
# _handle_401 raises ProviderError — this line never runs
|
||||
# ── Auth failure: token expired / invalid ─────────────────
|
||||
# Not keyed on 401 alone: a provider is free to answer an
|
||||
# invalid session with some other status, and claude.ai does
|
||||
# (403). Ask the provider rather than assuming.
|
||||
if self._is_auth_failure(response):
|
||||
self._handle_auth_failure(response)
|
||||
# _handle_auth_failure raises — this line never runs
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
f"{method} {url}",
|
||||
RuntimeError("401 Unauthorized"),
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} — not authenticated"
|
||||
),
|
||||
)
|
||||
|
||||
# ── 429: rate limited ──────────────────────────────────────
|
||||
@@ -476,11 +482,22 @@ class BaseProvider(ABC):
|
||||
last_exc or RuntimeError("Unknown error"),
|
||||
)
|
||||
|
||||
def _handle_401(self) -> None:
|
||||
"""Log a clear human-readable message for a 401 and raise ProviderError."""
|
||||
def _is_auth_failure(self, response: Any) -> bool:
|
||||
"""Whether this response means the stored credential is not valid.
|
||||
|
||||
Defaults to 401. Providers whose API reports an invalid session with a
|
||||
different status override this — claude.ai returns 403, so keying auth
|
||||
handling on 401 alone reports an expired key as a generic permission
|
||||
error and never tells the user to refresh it.
|
||||
"""
|
||||
return bool(response.status_code == 401)
|
||||
|
||||
def _handle_auth_failure(self, response: Any) -> None:
|
||||
"""Log a clear human-readable message for an auth failure and raise."""
|
||||
# Subclasses override to include provider-specific cookie name
|
||||
msg = (
|
||||
f"[{self.provider_name}] Authentication failed (401 Unauthorized). "
|
||||
f"[{self.provider_name}] Authentication failed "
|
||||
f"(HTTP {response.status_code}). "
|
||||
"Your session token has likely expired. "
|
||||
"Run 'ai-chat-exporter auth' to refresh your token."
|
||||
)
|
||||
@@ -488,7 +505,9 @@ class BaseProvider(ABC):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
"authentication",
|
||||
RuntimeError("401 Unauthorized — token expired"),
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} — session token expired or invalid"
|
||||
),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
|
||||
@@ -307,9 +307,9 @@ class ChatGPTProvider(BaseProvider):
|
||||
)
|
||||
return access_token
|
||||
|
||||
def _handle_401(self) -> None:
|
||||
def _handle_auth_failure(self, response: Any) -> None:
|
||||
msg = (
|
||||
"[chatgpt] Authentication failed (401 Unauthorized). "
|
||||
f"[chatgpt] Authentication failed (HTTP {response.status_code}). "
|
||||
"Your __Secure-next-auth.session-token has likely expired (~7 day lifetime). "
|
||||
"The session token is used to obtain a short-lived access token via /api/auth/session. "
|
||||
"To refresh: open chatgpt.com in Chrome → F12 → Application → Cookies "
|
||||
@@ -320,7 +320,9 @@ class ChatGPTProvider(BaseProvider):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
"authentication",
|
||||
RuntimeError("401 Unauthorized — ChatGPT token expired"),
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} — ChatGPT session token expired"
|
||||
),
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
+40
-5
@@ -2,6 +2,7 @@
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
from curl_cffi import requests as curl_requests
|
||||
|
||||
@@ -38,7 +39,9 @@ class ClaudeProvider(BaseProvider):
|
||||
Cloudflare's bot detection (same issue as chatgpt.com).
|
||||
|
||||
Authentication: sessionKey cookie (~30 day lifetime, opaque string).
|
||||
Expiry cannot be decoded client-side — a 401 is the only signal.
|
||||
Expiry cannot be decoded client-side, so an API rejection is the only
|
||||
signal — and claude.ai sends 403 permission_error / account_session_invalid
|
||||
for an invalid session, not 401. See ``_is_auth_failure``.
|
||||
"""
|
||||
|
||||
provider_name = "claude"
|
||||
@@ -71,11 +74,41 @@ class ClaudeProvider(BaseProvider):
|
||||
self._org_id: str | None = None # cached per session
|
||||
logger.debug("[claude] Session initialised with Chrome TLS impersonation (key: [REDACTED])")
|
||||
|
||||
def _handle_401(self) -> None:
|
||||
# claude.ai answers an invalid or expired sessionKey with 403
|
||||
# permission_error / account_session_invalid — never 401. Verified live
|
||||
# 2026-09-20 against GET /api/organizations: a valid key returns 200, while
|
||||
# an expired key, a garbage key and no cookie at all return byte-identical
|
||||
# 403s carrying this code. Matching on the code rather than on the bare
|
||||
# status keeps a genuine permission problem (which would carry a different
|
||||
# code) reported as itself.
|
||||
_SESSION_INVALID_CODE = "account_session_invalid"
|
||||
|
||||
def _is_auth_failure(self, response: Any) -> bool:
|
||||
if response.status_code == 401:
|
||||
return True
|
||||
if response.status_code != 403:
|
||||
return False
|
||||
try:
|
||||
body = response.json()
|
||||
except Exception:
|
||||
return False
|
||||
if not isinstance(body, dict):
|
||||
return False
|
||||
error = body.get("error")
|
||||
if not isinstance(error, dict):
|
||||
return False
|
||||
details = error.get("details")
|
||||
if not isinstance(details, dict):
|
||||
return False
|
||||
return details.get("error_code") == self._SESSION_INVALID_CODE
|
||||
|
||||
def _handle_auth_failure(self, response: Any) -> None:
|
||||
msg = (
|
||||
"[claude] Authentication failed (401 Unauthorized). "
|
||||
f"[claude] Authentication failed (HTTP {response.status_code}). "
|
||||
"Your sessionKey has likely expired (~30 day lifetime). "
|
||||
"Note: Claude session keys are opaque — a 401 is the only expiry signal. "
|
||||
"Note: Claude session keys are opaque, and claude.ai reports an "
|
||||
"invalid one as 403 'Invalid authorization', not 401 — an API "
|
||||
"rejection is the only expiry signal. "
|
||||
"To refresh: open claude.ai in Chrome → F12 → Application → Cookies "
|
||||
"→ find 'sessionKey' → copy the value. "
|
||||
"Then run 'ai-chat-exporter auth' or update CLAUDE_SESSION_KEY in .env."
|
||||
@@ -84,7 +117,9 @@ class ClaudeProvider(BaseProvider):
|
||||
raise ProviderError(
|
||||
self.provider_name,
|
||||
"authentication",
|
||||
RuntimeError("401 Unauthorized — Claude session key expired"),
|
||||
RuntimeError(
|
||||
f"HTTP {response.status_code} — Claude session key expired or invalid"
|
||||
),
|
||||
)
|
||||
|
||||
def _get_org_id(self) -> str:
|
||||
|
||||
@@ -118,7 +118,10 @@ _HARNESS_TAG_RE = re.compile(
|
||||
r"|<command-args>.*?</command-args>"
|
||||
r"|<command-contents>.*?</command-contents>"
|
||||
r"|<local-command-stdout>.*?</local-command-stdout>"
|
||||
r"|<system-reminder>.*?</system-reminder>",
|
||||
r"|<system-reminder>.*?</system-reminder>"
|
||||
# A fork subagent's first user turn: generic worker rules ahead of the
|
||||
# fork's actual "Your directive: …", which is kept.
|
||||
r"|<fork-boilerplate>.*?</fork-boilerplate>",
|
||||
re.DOTALL,
|
||||
)
|
||||
|
||||
@@ -530,6 +533,7 @@ def _extract_messages(
|
||||
policy: str,
|
||||
subagents: dict | None = None,
|
||||
include_sidechain: bool = False,
|
||||
expanding: frozenset[str] = frozenset(),
|
||||
) -> list[dict]:
|
||||
"""Normalize Claude Code records into messages.
|
||||
|
||||
@@ -540,6 +544,13 @@ def _extract_messages(
|
||||
those recursive subagent passes (subagent records are flagged
|
||||
``isSidechain``); the top-level pass keeps skipping sidechain records so a
|
||||
subagent is never also emitted as a stray top-level turn.
|
||||
|
||||
``expanding`` holds the spawn ids of the subagents being folded around
|
||||
this pass. A ``fork`` subagent's transcript opens with a copy of the parent
|
||||
turn that spawned it, its own spawn call included; that copy is dropped,
|
||||
since the enclosing subagent block already stands for it. Expanding it
|
||||
again recursed without end (RecursionError, every daily sync from
|
||||
2026-09-24).
|
||||
"""
|
||||
subagents = subagents or {}
|
||||
messages: list[dict] = []
|
||||
@@ -620,6 +631,8 @@ def _extract_messages(
|
||||
elif item_type == "tool_use":
|
||||
name = item.get("name") or "tool"
|
||||
tool_id = item.get("id")
|
||||
if tool_id in expanding:
|
||||
continue
|
||||
if name in _SUBAGENT_TOOL_NAMES and tool_id in subagents:
|
||||
# A Task/Agent spawn: fold its separate transcript inline
|
||||
# instead of collapsing it. Its own tool traffic is
|
||||
@@ -633,6 +646,7 @@ def _extract_messages(
|
||||
policy,
|
||||
subagents,
|
||||
include_sidechain=True,
|
||||
expanding=expanding | {tool_id},
|
||||
)
|
||||
blocks.append(
|
||||
make_subagent_block(
|
||||
|
||||
@@ -370,6 +370,59 @@ class TestSubagentFold:
|
||||
]
|
||||
assert "stray sidechain" not in top_text
|
||||
|
||||
def test_fork_containing_its_own_spawn_call(self, tmp_path):
|
||||
# The shape Claude Code writes for a `fork` subagent: a context-ref
|
||||
# record, a copy of the parent turn holding the fork's own spawn call,
|
||||
# then the directive behind <fork-boilerplate>. Folding that copied
|
||||
# call recursed until RecursionError.
|
||||
fork_records = [
|
||||
{"type": "fork-context-ref", "agentId": "a1", "parentLastUuid": "u0"},
|
||||
{
|
||||
"type": "assistant", "isSidechain": True,
|
||||
"message": {"role": "assistant", "content": [
|
||||
{"type": "tool_use", "id": "toolu_sub1", "name": "Agent",
|
||||
"input": {"description": "research"}},
|
||||
]},
|
||||
},
|
||||
{
|
||||
"type": "user", "isSidechain": True,
|
||||
"timestamp": "2026-05-01T10:00:06.000Z",
|
||||
"message": {"role": "user", "content": [
|
||||
{"type": "tool_result", "tool_use_id": "toolu_sub1",
|
||||
"content": [{"type": "text", "text": "Fork started"}]},
|
||||
{"type": "text", "text":
|
||||
"<fork-boilerplate>\nYou are a worker fork.\n"
|
||||
"</fork-boilerplate>\n\nYour directive: research X"},
|
||||
]},
|
||||
},
|
||||
{
|
||||
"type": "assistant", "isSidechain": True,
|
||||
"timestamp": "2026-05-01T10:00:07.000Z",
|
||||
"message": {"role": "assistant", "content": [
|
||||
{"type": "text", "text": "Fork result."},
|
||||
]},
|
||||
},
|
||||
]
|
||||
f = _write_session(tmp_path, _parent_with_task(), name="parent-3")
|
||||
_write_subagent(
|
||||
f, "toolu_sub1", fork_records,
|
||||
agent_type="fork", description="research X",
|
||||
)
|
||||
p = self._provider(tmp_path)
|
||||
p.fetch_all_conversations()
|
||||
conv = p.normalize_conversation(p.get_conversation("parent-3"))
|
||||
|
||||
subs = [
|
||||
b for m in conv["messages"] for b in m["blocks"] if b["type"] == "subagent"
|
||||
]
|
||||
assert len(subs) == 1
|
||||
inner = [bb for m in subs[0]["messages"] for bb in m["blocks"]]
|
||||
assert not any(b["type"] == "subagent" for b in inner)
|
||||
inner_text = [b.get("text") for b in inner]
|
||||
assert "Your directive: research X" in inner_text
|
||||
assert "Fork result." in inner_text
|
||||
assert not any("worker fork" in (t or "") for t in inner_text)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Repo tags in title (Part C)
|
||||
|
||||
@@ -1184,6 +1184,133 @@ class TestErrorBodyDiagnostics:
|
||||
assert "[REDACTED]" in described
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auth failure detection: claude.ai reports an invalid/expired sessionKey as
|
||||
# 403 permission_error / account_session_invalid, never 401. Verified live
|
||||
# 2026-09-20 against GET /api/organizations — a valid key returned 200, while
|
||||
# an expired key, a garbage key and no cookie at all returned byte-identical
|
||||
# 403s. Keying auth handling on 401 alone hid the refresh instructions behind
|
||||
# a raw JSON dump.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestAuthFailureDetection:
|
||||
class _Resp:
|
||||
reason = ""
|
||||
headers: dict = {}
|
||||
|
||||
def __init__(self, status, payload=None):
|
||||
self.status_code = status
|
||||
self.ok = 200 <= status < 400
|
||||
self._payload = payload
|
||||
self.text = ""
|
||||
|
||||
def json(self):
|
||||
if self._payload is None:
|
||||
raise ValueError("not json")
|
||||
return self._payload
|
||||
|
||||
@staticmethod
|
||||
def _bare(cls, response):
|
||||
p = cls.__new__(cls)
|
||||
p._request_delay = 0
|
||||
p._last_request_at = None
|
||||
p._session = type("S", (), {"request": lambda *a, **k: response})()
|
||||
return p
|
||||
|
||||
@staticmethod
|
||||
def _session_invalid(message="Invalid authorization"):
|
||||
return {
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "permission_error",
|
||||
"message": message,
|
||||
"details": {
|
||||
"error_code": "account_session_invalid",
|
||||
"error_visibility": "user_facing",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
def test_claude_403_session_invalid_is_an_auth_failure(self):
|
||||
from src.providers.base import ProviderError
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
resp = self._Resp(403, self._session_invalid())
|
||||
prov = self._bare(ClaudeProvider, resp)
|
||||
assert prov._is_auth_failure(resp) is True
|
||||
|
||||
with pytest.raises(ProviderError) as exc:
|
||||
prov._make_request("GET", "https://claude.ai/api/organizations")
|
||||
# The actionable message, not a dump of the response body.
|
||||
assert exc.value.operation == "authentication"
|
||||
assert "session key expired or invalid" in str(exc.value.original)
|
||||
|
||||
def test_claude_403_auth_failure_tells_the_user_how_to_refresh(self, caplog):
|
||||
from src.providers.base import ProviderError
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
resp = self._Resp(403, self._session_invalid())
|
||||
with caplog.at_level(logging.ERROR):
|
||||
with pytest.raises(ProviderError):
|
||||
self._bare(ClaudeProvider, resp)._make_request(
|
||||
"GET", "https://claude.ai/api/organizations"
|
||||
)
|
||||
logged = caplog.text
|
||||
assert "sessionKey" in logged
|
||||
assert "CLAUDE_SESSION_KEY" in logged
|
||||
# States the status it actually saw, rather than claiming 401.
|
||||
assert "403" in logged
|
||||
assert "401 Unauthorized" not in logged
|
||||
|
||||
def test_claude_403_with_another_error_code_is_not_an_auth_failure(self):
|
||||
"""A genuine permission problem must stay reported as itself."""
|
||||
from src.providers.base import ProviderError
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
payload = {
|
||||
"type": "error",
|
||||
"error": {
|
||||
"type": "permission_error",
|
||||
"message": "Organization access denied",
|
||||
"details": {"error_code": "org_access_denied"},
|
||||
},
|
||||
}
|
||||
resp = self._Resp(403, payload)
|
||||
prov = self._bare(ClaudeProvider, resp)
|
||||
assert prov._is_auth_failure(resp) is False
|
||||
|
||||
with pytest.raises(ProviderError) as exc:
|
||||
prov._make_request("GET", "https://claude.ai/api/organizations")
|
||||
assert exc.value.operation != "authentication"
|
||||
assert "Organization access denied" in str(exc.value.original)
|
||||
|
||||
def test_claude_403_with_unparseable_body_is_not_an_auth_failure(self):
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
resp = self._Resp(403, None)
|
||||
assert self._bare(ClaudeProvider, resp)._is_auth_failure(resp) is False
|
||||
|
||||
def test_claude_401_is_still_an_auth_failure(self):
|
||||
from src.providers.claude import ClaudeProvider
|
||||
|
||||
resp = self._Resp(401, {})
|
||||
assert self._bare(ClaudeProvider, resp)._is_auth_failure(resp) is True
|
||||
|
||||
def test_chatgpt_403_is_not_an_auth_failure(self):
|
||||
"""Guards the deleted-asset path: a media 403 is not an expired token."""
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
|
||||
resp = self._Resp(403, {"detail": "Forbidden"})
|
||||
assert self._bare(ChatGPTProvider, resp)._is_auth_failure(resp) is False
|
||||
|
||||
def test_chatgpt_401_is_an_auth_failure(self):
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
|
||||
resp = self._Resp(401, {})
|
||||
assert self._bare(ChatGPTProvider, resp)._is_auth_failure(resp) is True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Deleted assets: ChatGPT's download endpoint answers a missing upload with
|
||||
# 403 Forbidden, not 404. Measured 2026-08-17 over 18 such assets — every one
|
||||
|
||||
Reference in New Issue
Block a user