The render check settled it. Scrolling the whole conversation found sets of
images that do NOT display in ChatGPT — a set of 6 and a set of 4 — and they
are exactly our failures: the 6 are the records that 404 outright, the 4 are
the refused attachment batch we dumped (image.png, image(1).png,
image(2).png + 1). Every set that displays downloaded fine.
So the 403 was never OpenAI withholding something. It is the same failure
their own UI hits. All 19 are unrecoverable.
What the investigation established, now recorded in media.py and the
changelog so it is not repeated:
- uploads do not expire (36/36 sampled, 2025-09 → 2026-08, still live), so
export cadence was never the variable
- 7 records 404; 12 report state=ready with a library_file_id
- /files/{id}/download mints the signed estuary/content URL the UI fetches,
and refuses the survivors regardless of headers, Authorization, gizmo_id,
conversation_id, Referer or namespace; the Library id is file_not_found
- all survivors were created 2026-07-14 within minutes of each other yet
appear in conversations predating that date — a Library migration that
kept the metadata and lost the bytes
Dropping the nine one-off probes; their findings live in the code comments
and changelog, and git history has the scripts if they are ever wanted.
The decisive fact arrived: the images DO render in ChatGPT. So a valid
signature exists for files that hand us 403, and there is a route to find.
The rest is now pinned down. /files/{id}/download is the minting endpoint —
a working file's download_url is the same estuary/content URL the browser
uses. Signatures are per-file: swapping an id into a working URL returns
"Invalid signature or expired URL", and a ts without a sig gets the same.
So the difference is in the request, not the route.
Our call sends Authorization: Bearer plus cookies. A browser leans on
cookies and adds client headers a gated endpoint may check. This walks
those variants — no Authorization, conversation Referer, oai-device-id,
oai-language, sec-fetch-*, an image Accept, a conversation_id param —
against a file that is currently refused, and reports which mints a URL.
If none do, it prints the DevTools recipe for finding the minting call by
searching the Network panel for the file id.
Confirmed by the last run: a working file's download_url is
chatgpt.com/backend-api/estuary/content?cid&id&p&sig&ts&v — the same route
the browser uses. So /files/{id}/download is the minting endpoint, and a
gizmo file's 403 is a refusal to mint the signature. That is why no amount
of scoping helped; we were turned away at the only door that issues them.
Nothing else in the estuary namespace serves files: 404 across the board.
But section B tested file_000000001b3071f5…, one of the seven DELETED
files, because it took the first failure without checking its class. Those
results say nothing about the eleven recoverable ones.
- Pick the target by probing /files/{id} and taking one that answers 200,
skipping (and naming) the deleted ones.
- Add two experiments: estuary/content with a ts but no sig, since
validation asked only for id/p/ts and never sig; and a working file's
freshly minted URL with the refused id swapped in, which shows whether
the signature is bound to the file.
A URL copied from the browser gave us the route we never tried:
/backend-api/estuary/content?id=…&ts=496382&p=fs&cid=1&sig=…&v=0
Fetched with no cookies it returns 403 {"detail":"File stream access
denied."}, so the signature rides on the session rather than replacing it.
Every earlier probe lived under /backend-api/files/*; estuary/* is new
ground.
estuary_probe.py checks two things:
A. What /files/{id}/download hands back for a file that works. If its
download_url is an estuary URL, that endpoint is the minting step, and a
gizmo file's 403 is a refusal to mint — which is why no amount of
scoping helped.
B. Whether the estuary namespace exposes a route that serves a refused
file directly.
It also replays a pasted URL through the exporter's session, and says
plainly whether the id in that URL is one of the refused files — the two
captured so far were working images, so they showed the shape without
telling us whether the broken ones have a URL at all.
The endpoint hunt is done guessing. Results:
/files/{libfile}/download 200 {"error_code":"file_not_found",
"error_type":"GetDownloadLinkError"}
— twice, on two different Library ids, so the
Library id is simply not valid there
/library/* 404 across every shape
POST on the download route 405
/content?asset_pointer= 422 requiring query params id, ts, p
That last one is the find: /backend-api/content is a signed route, and the
web UI has the values we cannot compute. So the question shifts from "which
endpoint" to "is the UI's URL reusable outside the browser".
try_pasted_url.py takes a URL copied from DevTools and fetches it three
ways — bare, through the exporter's authenticated session, and with the
Authorization header removed in case bearer and signature conflict. The
pattern says whether the exporter can fetch these at all, and therefore
whether it is worth hunting for what mints the signature.
Before any of that: check whether the images still render in ChatGPT's own
UI. If they show broken there, file_not_found is literally true, these 11
are lost like the other 7, and no route exists to find.
The pairing came back perfectly clean, and it is the whole diagnosis:
has library_file_id → refused (11/11)
no library_file_id → deleted (7/7)
And /files/{libfile_id}/download answered 200 — not 403, not 404 — with
{status, error_code, error_type, error_message}. That endpoint accepts the
Library id; it is returning an application-level error inside an HTTP 200.
The probe printed only the keys and dropped the message, which is the one
thing that says what the call is missing.
- show() now prints the full body for every response, and detects a
non-JSON 200 as possible raw bytes.
- Added routes worth ruling in or out: the same call as POST, with a
conversation_id, /files/{lib}/content, /content?asset_pointer=, and four
Library listing endpoints — a listing usually reveals both the id form
the UI uses and the route that actually serves bytes.
- Probes a second Library id too, so one odd file cannot mislead us.
Dumping the raw message metadata found the identity the asset pointer never
carried:
"id": "file_000000003454722f9481506b96aed510" ← refused
"library_file_id": "libfile_4eb82f478fe081919127e2eba9886e86"
"source": "local"
The exporter only knows the sediment id from the asset_pointer and asks
/files/{sediment}/download, which 403s for these. The Library is a separate
store with its own ids, so we have been asking for the conversation-scoped
copy of a file that now lives in the Library. That also fits the 2026-07-14
creation-time cluster: a Library migration would mint exactly these new
records.
Neither gizmo_id, conversation_id, a /gizmos path nor a project Referer
helped (403/404), so scope was never the missing piece — identity was.
library_download_probe.py pairs each refused sediment id with its
library_file_id from message.metadata.attachments and tries the endpoints
that could serve it.
metadata_diff found the discriminator. Every refused image carries
use_case="gizmo" — ChatGPT's name for Projects and custom GPTs — while
everything that downloads is image_gen or multimodal. All are state=ready,
so nothing is damaged: the plain /files/{id}/download endpoint just will
not serve a project-scoped file.
Second clue: all 11 were created 2026-07-14 within minutes of each other,
yet appear in conversations dated 2025-11 through 2026-08, several
predating their own creation_time. Something on July 14 re-created them as
gizmo-scoped copies and repointed the conversations — which is also why
the losses start in July. Not a policy change, an event.
gizmo_download_probe.py dumps the raw conversation part carrying one of
these assets (it may simply name the scope the download wants) and then
tries the plausible calls — gizmo_id/conversation_id query params, a
/gizmos/{id}/files path, a project Referer. A 200 is the fix.
branch_check killed the abandoned-branch hypothesis — every lost image is
on the live branch, and the 3 images that do sit on abandoned branches are
alive. But it turned up something the earlier probing missed by sampling
four IDs from one family and generalising: of the 19 failures, only 7 are
actually gone. The other 12 answer /files/{id} with 200 and full metadata
and refuse only /download. They exist, and may be recoverable.
Three states, then: gone (404), refused (200 meta + 403 download), working
(200 both). Since metadata comes back for the refused ones, the
discriminator can be read straight off — fetch it for every file in each
state and compare fields, flagging any field whose values never overlap
between states.
Also re-checks /download now: if a file refused during the export serves
today, those 403s were transient and a retry pass recovers them, which is
a completely different fix from anything permanent.
Expiry is now ruled out: 36/36 sampled images from 2025-09 through
2026-08 are still live server-side, so nothing dies of age and export
cadence is not the variable. The loss is per-asset — 2026-07-09 kept 38
images and lost 12 in one conversation.
Next hypothesis: those images belong to messages that were edited or
regenerated. ChatGPT stores a conversation as a tree and editing forks it,
stranding the superseded messages on an abandoned branch. The exporter
walks every node (chatgpt.py:963), so it exports those branches too — and
an attachment unreachable from live history is a natural GC target.
branch_check.py fetches the raw conversation, derives the live branch by
walking parent links up from current_node, and cross-tabulates every image
against (on the live branch?) x (still downloadable?). If the lost images
are all off-branch and nothing on the live branch is missing, this is not
data loss at all — it is attachments to messages that were replaced.
analyze_media_age.py claimed age was ruled out because an image from
2025-11 was saved while one from 2026-08 was not. That conclusion does not
follow. "Saved" means some earlier run downloaded it, not that ChatGPT
still holds it — an image captured in June looks saved forever after, even
if it died in July. A month with no losses shows the exports were timely,
not that the assets survived.
- probe_survival_by_month.py: sample images already on disk, grouped by
their conversation's month, and ask /files/{id} whether each still
exists today. Old months still 200 → no expiry, and cadence did not save
them. Old months now 404 → uploads do expire and cadence is the whole
ballgame.
- analyze_media_age.py: stop asserting the unsupported verdict; say what
the number does and does not show, and point at the probe.
One signal there is immune to the confound and survives: 2026-07-09 kept
38 images and lost 12. Same conversation, same day, opposite outcomes —
no retention policy does that, so at least part of this is per-asset.
"Do I have to export within N days?" is answerable from the exports
already on disk — the renderer records every image's outcome inline
( when saved, a placeholder when not) and the
conversation date is in the filename. Group by month and source and the
hypotheses separate: a clean old/new cutoff means expiry, user_upload
dying at an age model_generated survives means the source matters, and
losses scattered through months that otherwise downloaded fine means
neither.
Offline, no token, no API calls.
The improved error reporting landed, and the answer it produced is
{"detail":"Forbidden"} — generic, no reason. The cause has to be narrowed
by experiment instead, so collect the experiments in one script:
- classify the failed assets offline from the exported placeholders
(user_upload vs model_generated) — no API call needed
- probe a known-good asset in the same session as a control, to rule the
session in or out
- retry the 403 with ChatGPT-Account-Id, which the exporter never sends
and which workspace-scoped resources can require
- compare /files/{id} against /files/{id}/download
Temporary: delete once the cause is known, or fold into `doctor` if the
check earns a permanent home.
Media downloads logged "HTTP Error 403:" with no reason. That string is
curl_cffi's raise_for_status() format, "HTTP Error {code}: {reason}", and
HTTP/2 carries no reason phrase — so the message said nothing, and
_make_request threw the response body away. The provider's JSON `detail`
is the only explanation available for a refused asset.
- base._make_request: end non-retryable statuses with a ProviderError
carrying the body's detail/error/message (redacted, truncated to 300
chars) instead of a bare raise_for_status().
- media: bucket 403 as `forbidden` in the run summary, separately from
`download-error` — "the asset is gone" and "we were refused" are
different problems.
- utils.redact_secrets: match secret key names per word. Exact matching
let access_token, api_key, and session-token through into logged
bodies; "keywords"/"monkey"/"tokenizer" stay intact.
- tests/test_config.py: test_defaults depended on the absence of a local
.env — load_config() calls load_dotenv(override=False), which restored
the variable the test had just deleted. Stub dotenv discovery.
305 tests pass.
- Subagents: fold Task-tool transcripts (subagents/*.jsonl) inline as
collapsible <details> blocks at the spawn point; their own tool traffic
collapses under the same policy; nesting handled via toolUseId matching
- Joplin: Claude Code gets its own top-level AI-ClaudeCode notebook;
update_note sets parent_id so notes self-heal/relocate on re-sync
- Repo [tags] in titles via git-root detection (nearest .git ancestor),
home-wide and cross-workspace; CLAUDE_CODE_REPO_TAG_IGNORE escape hatch
- Multi-root scanning: CLAUDE_CODE_DIR as os.pathsep list + CLAUDE_CONFIG_DIR;
merged by launch-folder, newer-mtime wins on UUID collision
- 298 tests passing
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mark FUTURE.md as feature-complete as of v0.7.0: active roadmap is empty and
the deprioritized backlog (Joplin --force, per-conversation cache reset,
official export-ZIP fallback, o1/o3 reclassification, Obsidian output,
token-expiry notifications, search) is closed as not needed, kept for
reference only. No further work planned.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The export reads ChatGPT's and Claude's undocumented internal web APIs,
which can change shape without notice; the worst failure for a backup tool
is a silent one (skipped/mis-parsed content with no error). Add a `canary`
command + BaseProvider.check_drift() (overridden by ChatGPT/Claude) that
fetches one listing page + one conversation per provider and asserts only
the normalizer's load-bearing fields — not the full response shape, which
churns harmlessly. The top silent risk it guards is a renamed retrieval-tool
author bypassing the hidden-content collapse. ERROR findings exit non-zero;
WARN findings are surfaced but non-fatal so a backup run is never blocked.
Also retires the in-app watch mode and headless StartOS direction from the
roadmap (tool stays a local, manually-run CLI) and updates docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ChatGPT session tokens are JWEs whose `exp` is encrypted and unreadable
client-side, so the old JWT-decode path could never yield an expiry, and the
`/api/auth/session` `expires` is a misleading rolling window (it advances on
every call even for a dead token). The honest signal is the `error` field:
`RefreshAccessTokenError` means the session token is dead while `expires` and
a stale `accessToken` are still echoed (verified live).
- Add ChatGPTProvider.session_health() and a "ChatGPT token active" doctor
check based on it; drop the dead JWE/exp decode path in doctor and auth.
- Fix _fetch_access_token to fail fast on a set `error` instead of returning
the stale accessToken (which produced a confusing downstream 401).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Browser cookie auto-extraction is not viable: modern Chromium App-Bound
Encryption (Chrome 127+/current Brave) keys cookies off a SYSTEM-level
layer that cannot be decrypted off disk without admin rights and AV-flagged
SYSTEM impersonation, and fails on Brave specifically. Drop the
`--from-browser` flag, `_auth_from_browser`, `src/browser_tokens.py`, its
test, and the browser-cookie3 dependency. Auth is manual (DevTools wizard)
only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mark_exported() was discarding created_at from the metadata dict because
it wasn't in the hardcoded stored-key list, so the joplin sync always
saw an empty date and omitted the prefix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Joplin notebooks now use a two-level hierarchy: AI-ChatGPT / <project> and
AI-Claude / <project> instead of a single flat title. Note titles are prefixed
with the conversation created_at date (YYYY-MM-DD). Export folders collapse
provider/project/year into a single provider/project.year directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
First real-data export against v0.4.0 surfaced 66 unknown blocks across
three content types — captured live and added.
Added:
- execution_output (Code Interpreter / container.exec / python tool
output) → tool_result block. output=content.text,
tool_name=author.name, is_error=metadata.aggregate_result.status,
summary=metadata.reasoning_title
- system_error → error tool_result with tool_name=author.name
- tether_browsing_display: spinner placeholders (empty result+summary)
skip silently with DEBUG log; defensive populated-case branch maps
to tool_result (untested in real data)
- tool_result block schema: optional `summary` field rendered as
italic line between header and fence
- tool_result rendering: tool_name appears in header when present
(e.g. `📤 Result: container.exec`); existing tool_name=None calls
unchanged
- _ROLE_LABELS["tool"] = ("🔧 Tool", "tool")
Fixed:
- chatgpt.normalize_conversation reads `conversation_id` as fallback
for `id`. Live API uses conversation_id; fixtures use id.
Pre-fix: empty id in YAML frontmatter and missing context in
WARNING logs.
Tests: 11 new (192 total, 0 failures). Fixture extended with 4
tool-output cases (execution_output success, empty execution_output
that should skip, system_error, tether_browsing_display spinner).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extracts per-message content into a typed `blocks` list (text, code,
thinking, tool_use, tool_result, image_placeholder, file_placeholder,
unknown) and renders them at exporter write time. Voice transcripts,
Custom Instructions, and image references now appear in exports
instead of being silently dropped.
Foundation:
- src/blocks.py: pure block constructors, _safe_fence (fence-corruption
defense, verified live in Joplin), _blockquote_prefix, render
- src/loss_report.py: per-run tally surfaced as INFO summary at end of
export so silently-dropped data becomes visible
Providers:
- ChatGPT: dispatch on content_type produces typed blocks; voice shapes
(audio_transcription, audio_asset_pointer, real_time_user_audio_video_
asset_pointer) locked from live DevTools capture; Custom Instructions
bug fix (parts-vs-direct-fields); role filter lifted; hidden-context
marker driven by is_visually_hidden_from_conversation flag
- Claude: defensive dispatch for text/thinking/tool_use/tool_result/image
with recursive nested-block flattening; untested against real rich-
content data — fix-forward in v0.4.1
Exporter:
- Markdown renders from blocks at write time via render_blocks_to_markdown;
backward-compat fallback to content for any pre-v0.4.0 cached data
Tests:
- 27 new tests across providers, exporters, CLI; fixtures rebuilt with
real-shape ChatGPT voice + Custom Instructions cases
- 181/181 pass
Behavior changes (intentional):
- JSON output omits content; consumers should read blocks
- Per-conversation message counts increase (Custom Instructions, image-
only, tool-only messages now appear)
- Existing exports not auto-re-rendered; users wanting fresh output run
cache --clear then export
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Support __Secure-next-auth.session-token.0/.1 split cookies; ChatGPT
now issues tokens that exceed the 4KB per-cookie limit and must be
sent as two named chunks or the auth endpoint returns no accessToken.
Add CHATGPT_SESSION_TOKEN_1 env var; update auth wizard instructions.
- Fix Claude conversations exported to wrong directory when project name
is present in the listing but absent from the detail endpoint response.
Explicitly propagate "project" alongside _-prefixed annotation keys.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude's list endpoint returns conversations with a `name` field rather
than `title`, so every Claude row was falling through to "Untitled".
Also set no_wrap + ellipsis overflow and tune column widths so the table
renders one row per conversation in Windows Command Prompt (80 cols).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Core features:
- Add `joplin` command: syncs exported Markdown to Joplin via local REST API
- Notebooks auto-created per provider+project (e.g. "ChatGPT - My Project")
- Idempotent: notes updated (not duplicated) on re-run; note ID tracked in manifest
- Add `--project` filter to `export` and `list` commands (substring or 'none')
- Add ChatGPT Projects support via CHATGPT_PROJECT_IDS env var
Config:
- Add JOPLIN_API_TOKEN, JOPLIN_API_URL, JOPLIN_REQUEST_TIMEOUT
- Version now read from importlib.metadata (single source of truth: pyproject.toml)
- Bump version to 0.2.0
Quality:
- Explicit Timeout handling in JoplinClient with actionable error messages
- token validation (validate_token) separate from connectivity (ping)
- Remove debug_auth.py, debug_claude.py, and untracked .har file
- Add *.har to .gitignore (may contain auth cookies/session tokens)
- Update README, CHANGELOG, FUTURE.md to reflect v0.2.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
claude.ai has the same Cloudflare TLS fingerprinting protection as
chatgpt.com. Apply the same fix: curl_cffi impersonate=chrome120,
remove base class User-Agent to avoid JA3/UA mismatch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
curl_cffi sets a User-Agent consistent with its JA3 TLS fingerprint.
BaseProvider's custom UA (Chrome/121) conflicted with the chrome120
TLS fingerprint, causing Cloudflare to flag the request as a bot.
Removing the UA from session headers lets curl_cffi manage its own.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
chatgpt.com uses Cloudflare's TLS fingerprinting (JA3/JA4) which
blocks Python requests regardless of cookies. curl_cffi impersonates
Chrome's exact TLS handshake, making requests indistinguishable from
a real browser at the transport layer.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Using self._session.cookies.set() ensures the cookie is sent correctly
by the requests session on all calls, including /api/auth/session.
Also add sec-fetch-* headers required by chatgpt.com.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The __Secure-next-auth.session-token cannot be used directly as a Bearer
token. It must first be exchanged via GET /api/auth/session (with the token
sent as a Cookie) to obtain a short-lived accessToken. This accessToken is
then used as the Authorization: Bearer header for all backend-api calls.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Doctor was reading env vars before loading .env, so tokens set in .env
were invisible. ChatGPT now uses JWE (encrypted JWT) tokens which
PyJWT cannot decode without the server key — treat decode failure as
"token set, expiry unknown" rather than a FAIL.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
