release: v1.0.0, and split the README into docs/
The archive does what it was built to do, unattended, on both machines: four providers, incremental and resumable, Markdown plus Joplin, on a schedule, with results pushed to a phone and failures that announce themselves. The one item left on the roadmap — a StartOS service that consolidates both machines into one corpus — is an optional convenience, not a gap. Hence 1.0.0 rather than another 0.x. The code in this release shipped across b6ce636..8c3e00a: the claude.ai 403 session-invalid fix, the fork-subagent recursion fix, the FAILED-push backstops on both platforms, and asset-download retries with cross-run recovery, a circuit breaker and media counts in the notification. This commit cuts the release and pays a documentation debt. The README had reached 939 lines / 6,490 words / 48 KB / 96 headings, doing four unrelated jobs at once, and FUTURE.md asked for the split to ride along with the next substantial README edit rather than break every existing anchor for its own sake. A release is that edit. README.md is now 303 lines and keeps only what you need to install and configure the tool; the rest moved verbatim into docs/providers.md, docs/cli.md, docs/scheduling.md and docs/troubleshooting.md. No section was rewritten or dropped and every original heading still exists, so each anchor still resolves somewhere — but bookmarks into moved sections now need the docs/ path, which is the cost this deferral was protecting against. .gitignore needed the fix FUTURE.md predicted: the blanket *.md rule (exported conversations are Markdown and may be private) matched all four new pages, and git check-ignore confirmed every one would have been silently ignored. !docs/*.md is added with a comment saying why. Gitea's rendering of relative docs/ links cannot be checked before pushing, so it still needs confirming after. Also corrects two stale claims in the changelog: the Windows backstop is no longer "not yet run under Windows PowerShell 5.1 or Task Scheduler" (confirmed on winmini today, LastTaskResult 0 — only the FAILED push itself remains unexercised there), and the test count is 414, not 409. 414 tests passing. No code changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012gCWSknsWvhLnMFwa7qnGp
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
8c3e00a520
commit
04ebb8987c
@@ -24,6 +24,9 @@ exports/
|
||||
!FUTURE.md
|
||||
!FUTURE-ARCHIVE.md
|
||||
!CHANGELOG.md
|
||||
# The split-out documentation (README.md's detail pages). Without this the
|
||||
# blanket *.md above would ignore every new docs/ page silently, with no error.
|
||||
!docs/*.md
|
||||
|
||||
# Cache and logs
|
||||
cache/
|
||||
|
||||
+25
-4
@@ -3,7 +3,17 @@
|
||||
All notable changes to this project will be documented here.
|
||||
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
|
||||
## [Unreleased]
|
||||
## [1.0.0] - 2026-10-20
|
||||
|
||||
The archive does what it was built to do, on both machines, unattended: four
|
||||
providers, incremental and resumable, Markdown plus Joplin, on a schedule, with
|
||||
results pushed to a phone and failures that announce themselves. What remains on
|
||||
the roadmap is one optional consolidation (`FUTURE.md` §1), not a gap — so this
|
||||
is 1.0.0 rather than another 0.x.
|
||||
|
||||
This release is mostly the unglamorous half of that claim: four bugs that each
|
||||
made a scheduled run lie about its own outcome, and the documentation split that
|
||||
939 lines had outgrown.
|
||||
|
||||
### Fixed
|
||||
- **A transient 5xx on an image download lost the image permanently, and a timeout lost the whole conversation.** An asset download is two hops: `GET /files/{id}/download` mints a signed URL, then that URL is fetched for the bytes. Only the first went through `_make_request`. The second was a bare `session.request` with no 5xx handling at all, which produced two distinct failures in the 2026-10-20 09:00 run:
|
||||
@@ -31,9 +41,20 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
- **A media circuit breaker, so a bad day at the provider costs one quiet morning.** Five consecutive *transient* asset failures stop downloads for the remainder of the run; the skipped assets are recorded as retryable rather than abandoned, so the next run collects them. Permanent failures never count toward tripping it — a deleted upload says nothing about the provider's health, and this archive holds 19 of those that would otherwise trip it on a perfectly healthy server. Any success resets the streak, so ordinary scattered losses never trip it either. Assets already on disk are still linked while the breaker is open, since that needs no network.
|
||||
- **Media failures appear in the push notification.** `format_summary` now adds `N media failed` per provider. The 2026-10-20 run lost two assets and pushed a clean "AI archive OK" with no hint of it — the counts existed only in the log file, which is somewhere you have to think to go and look. Counts only, never asset ids, for the same reason conversation titles are excluded: a topic on public ntfy.sh is readable by anyone who knows its name. Media failures deliberately do **not** raise the notification's priority or mark the run failed — the transcript is archived and a transient asset recovers by itself.
|
||||
|
||||
### Changed
|
||||
- **The README is split into a short overview plus four documents in `docs/`,** closing `FUTURE.md` §2. It had reached 939 lines / 6,490 words / 48 KB / 96 headings — a ~25-minute read doing four unrelated jobs at once, and still growing (it was 830 / 5,562 / 37 KB / 61 when the split was first proposed). `README.md` is now 303 lines and keeps only what you need to install and configure the tool: the ToS warning, installation, `doctor`, `.env` reference, output structure, security notes. The rest moved verbatim — no section was rewritten or dropped, and every original heading still exists, so every anchor still resolves *somewhere*:
|
||||
|
||||
| Document | Content |
|
||||
|----------|---------|
|
||||
| `docs/providers.md` | Session tokens, lifetimes and expiry detection, the `auth` wizard, ChatGPT Projects, Claude Code sessions, Codex sessions |
|
||||
| `docs/cli.md` | CLI reference — all eleven commands |
|
||||
| `docs/scheduling.md` | systemd timer, Windows task, what to know before relying on it, notifications |
|
||||
| `docs/troubleshooting.md` | How the cache works, every failure message, media download failures, filing a bug report |
|
||||
|
||||
The cost this deferral was protecting against is real and is now paid: a bookmark or note pointing at a README anchor that moved (`#cli-reference`, `#troubleshooting`, `#getting-your-session-tokens`, …) no longer resolves, because the anchor is in another file. The three questions `FUTURE.md` left open are settled as follows. **`.gitignore` did need the fix it predicted** — the blanket `*.md` rule (exported conversations are Markdown and may be private) matched all four new pages, and `git check-ignore` confirmed every one would have been committed-never and ignored silently; `!docs/*.md` is added alongside the existing per-file re-includes, with a comment saying why. **Cross-file anchors are used, not avoided**, but only for the four links that already existed between these sections, each verified to point at a heading that still exists. **Gitea rendering of relative `docs/` links is unverified** — it cannot be checked before pushing, so confirm it after.
|
||||
|
||||
### Notes
|
||||
- **`FUTURE.md` §2 (split the README) is still open and was deliberately not done here.** It asks to be folded into "the next substantial README edit", and this edit is a new subsection plus two cross-references — not substantial enough to justify breaking every existing link into a README anchor. The new material is written to survive the split: it lives under *Embedded images and media*, which that plan assigns to `docs/troubleshooting.md`.
|
||||
- Verified live rather than only in tests: the two assets lost in the morning run were re-downloaded successfully through the new path (`doctor` all green, 409 tests passing, a read-only `export --dry-run --provider chatgpt` exercising the refactored request core across 4 listing pages / 379 conversations). The cross-run retry and the breaker are covered by unit tests only — reproducing a multi-day storage incident against the live provider is not something this change waited for.
|
||||
- Verified live rather than only in tests: the two assets lost in the morning run were re-downloaded successfully through the new path (`doctor` all green, 414 tests passing, a read-only `export --dry-run --provider chatgpt` exercising the refactored request core across 4 listing pages / 379 conversations). The cross-run retry and the breaker are covered by unit tests only — reproducing a multi-day storage incident against the live provider is not something this change waited for.
|
||||
|
||||
- **Every scheduled Claude Code sync since 2026-09-24 died with `RecursionError`.** Claude Code's `fork` subagents write a transcript that opens with a copy of the parent turn that spawned them — the fork's own `Agent` call included. `_extract_messages` folds a subagent inline whenever it meets its spawn call, so it met that copy inside the fork, folded the same fork again, and recursed until Python's limit. One fork anywhere in `~/.claude/projects` was enough to fail the whole provider; seven existed across three sessions, and the Codex half of the run, unaffected, hid the cause behind a generic exit 1.
|
||||
|
||||
@@ -49,7 +70,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
|
||||
|
||||
The Windows task gets the same backstop. Its action is now `scheduling\run-sync.ps1` (via `powershell.exe -File`, window hidden) in place of the inline `cmd.exe` loop, with the same rules: every provider attempted, exit 1 if any failed, a FAILED push carrying the class only for a run that died unreported, `NTFY_NOTIFY=off` honoured, `NTFY_*` read from the environment first. Because Task Scheduler keeps no output, each run's output is appended to `cache\logs\scheduled-sync.log` (one rollover to `.1` past 1 MB), and the push points there. The wrapper sets `PYTHONIOENCODING=utf-8`: with stdout redirected to a file rather than a console, Python would otherwise encode in the ANSI code page and fail on the first `✓`. Re-run `Register-AiChatSyncTask.ps1` with your providers to pick it up.
|
||||
|
||||
Verified on Linux under PowerShell 7.6 with a stand-in `cmd.exe`, the same fake launcher and capture server: the same five outcomes as above, plus a bearer token, an environment `NTFY_TOPIC` overriding `.env`, an unreachable server (warning, exit still 1) and the `chatgpt,claude` single-argument form. Not yet run under Windows PowerShell 5.1 or Task Scheduler.
|
||||
Verified on Linux under PowerShell 7.6 with a stand-in `cmd.exe`, the same fake launcher and capture server: the same five outcomes as above, plus a bearer token, an environment `NTFY_TOPIC` overriding `.env`, an unreachable server (warning, exit still 1) and the `chatgpt,claude` single-argument form. Since confirmed on winmini under Windows PowerShell 5.1 and Task Scheduler (2026-10-20): `Register-AiChatSyncTask.ps1 -Provider chatgpt,claude` registered the wrapper, and a `Start-ScheduledTask` run of it completed with `LastTaskResult: 0`, writing both providers' output to `cache\logs\scheduled-sync.log`. Registering is a one-time write, so a `git pull` does **not** update an already-registered task — re-run the registration script after taking this change. The one path still unexercised on Windows is the FAILED push itself: it needs a run that dies without reporting, which has not happened there yet.
|
||||
|
||||
- **An expired Claude session key reported a raw JSON dump instead of how to fix it.** `_make_request` routed only **401** to the auth handler (`src/providers/base.py`), and claude.ai does not use 401 — an invalid or expired `sessionKey` comes back as `403 permission_error` with `details.error_code = account_session_invalid`. So the one message that names the cookie, its ~30-day lifetime and the DevTools path to refresh it could never fire for Claude. What the user got instead was the generic 4xx path: `HTTP 403 — error: {'type': 'permission_error', 'message': 'Invalid authorization'…}`, which reads like a permissions problem with the account and not like "your key expired, here is how to replace it."
|
||||
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Planned Future Work
|
||||
|
||||
> **Status 2026-08-18 (v0.9.0).** The tool archives four providers — two web
|
||||
> (`chatgpt`, `claude`) and two local agent-transcript (`claude-code`, `codex`)
|
||||
> — on a schedule, on Linux and Windows, reporting results by push notification.
|
||||
> Two items below are genuinely planned. Everything else has shipped or been
|
||||
> **Status 2026-10-20 (v1.0.0).** The tool archives four providers — two web (`chatgpt`,
|
||||
> `claude`) and two local agent-transcript (`claude-code`, `codex`) — on a
|
||||
> schedule, on Linux and Windows, reporting results by push notification.
|
||||
> **One item below is genuinely planned.** Everything else has shipped or been
|
||||
> decided against.
|
||||
|
||||
Completed work moves to the changelog; this file holds only what is *not* built
|
||||
@@ -71,42 +71,6 @@ What this needs, and what it does *not*:
|
||||
Meanwhile Joplin is sufficient — it already syncs (encrypted) offsite, and it is
|
||||
where the archive is actually read. This is a "nice eventually", not a gap.
|
||||
|
||||
## 2. Split the README into separate documents
|
||||
|
||||
The README is 830 lines / 5,562 words / 37 KB — about a 25-minute read, with 61
|
||||
headings. An H2-only table of contents was added 2026-08-18 and helps, but it
|
||||
treats the symptom: the file is doing at least four unrelated jobs at once.
|
||||
|
||||
Rough shape of a split:
|
||||
|
||||
| Document | Content today |
|
||||
|----------|---------------|
|
||||
| `README.md` | What it is, install, first run, a pointer to the rest |
|
||||
| `docs/providers.md` | ChatGPT Projects, Claude Code Sessions, Codex Sessions, session tokens |
|
||||
| `docs/cli.md` | CLI Reference — ~250 lines on its own, the single biggest section |
|
||||
| `docs/scheduling.md` | Scheduling a Daily Run, notifications |
|
||||
| `docs/troubleshooting.md` | Troubleshooting, How the Cache Works |
|
||||
|
||||
Not urgent, because it has a real cost the TOC did not: any existing link into a
|
||||
README section — a bookmark, a note, another repo, a commit message — breaks
|
||||
when that section moves to another file. Worth folding into the next substantial
|
||||
README edit rather than doing as a change of its own.
|
||||
|
||||
Two things to settle when it happens:
|
||||
|
||||
- **`.gitignore` ignores `*.md` on purpose** — exported conversations are
|
||||
Markdown and may contain private content — and re-includes each doc by name
|
||||
(`!README.md`, `!FUTURE.md`, …). New files under `docs/` will be silently
|
||||
ignored, with no error, until `!docs/*.md` is added. This already bit the
|
||||
creation of `FUTURE-ARCHIVE.md` on 2026-08-18.
|
||||
- Whether `docs/` renders acceptably on the Gitea instance hosting this repo.
|
||||
Relative links between Markdown files do work there, but confirm before
|
||||
splitting rather than after.
|
||||
- Whether anchors in the split files are stable enough to link *between*
|
||||
documents, or whether cross-references should point at file tops only. GFM
|
||||
anchors derive from heading text and break silently on a reword — the same
|
||||
fragility the TOC already carries.
|
||||
|
||||
---
|
||||
|
||||
# Decided against
|
||||
@@ -142,3 +106,4 @@ Detail for each release is in `CHANGELOG.md`.
|
||||
- **v0.7.0** — `canary` drift detection; real ChatGPT token-health check on `doctor`; removal of the non-viable browser-cookie path
|
||||
- **v0.8.0** — Claude Code coverage reopened: subagent capture as folded `<details>`, repo `[tags]` in titles, its own `AI-ClaudeCode` notebook with self-healing note moves, multi-root scanning (`CLAUDE_CODE_DIR`, `CLAUDE_CONFIG_DIR`)
|
||||
- **v0.9.0** — Codex CLI provider; launcher scripts removing the virtualenv ceremony on both platforms; `sync` with a meaningful exit code; daily scheduling for Linux and Windows; ntfy push notifications; ChatGPT project attribution via `gizmo_id` and the `projects` command; a silent data-loss fix in both local providers (`splitlines` breaking on U+0085/U+2028/U+2029)
|
||||
- **v1.0.0** — Reliability pass on unattended runs: asset downloads retried on 5xx with a per-asset time budget, transiently failed assets recovered on later runs, a media circuit breaker, media failures surfaced in the push notification; a timeout on an asset no longer aborts its conversation; claude.ai's 403 session-invalid recognised as an expired key; `fork` subagents no longer recurse forever; a crashed scheduled run pushes a FAILED notification on both platforms; the README split into an overview plus `docs/`
|
||||
|
||||
@@ -8,23 +8,26 @@ Supports incremental sync — only new or updated conversations are exported on
|
||||
|
||||
## Contents
|
||||
|
||||
**This page** — what it is, installing it, the first run, configuration and where files land.
|
||||
|
||||
- [Terms of Service Warning](#terms-of-service-warning)
|
||||
- [Installation](#installation)
|
||||
- [First Run: Run Doctor](#first-run-run-doctor)
|
||||
- [Getting Your Session Tokens](#getting-your-session-tokens)
|
||||
- [The `auth` Command](#the-auth-command)
|
||||
- [`.env` Setup](#env-setup)
|
||||
- [ChatGPT Projects](#chatgpt-projects)
|
||||
- [Claude Code Sessions](#claude-code-sessions)
|
||||
- [Codex Sessions](#codex-sessions)
|
||||
- [Scheduling a Daily Run](#scheduling-a-daily-run)
|
||||
- [Output Structure](#output-structure)
|
||||
- [CLI Reference](#cli-reference)
|
||||
- [How the Cache Works](#how-the-cache-works)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Documentation](#documentation)
|
||||
- [Future Work](#future-work)
|
||||
- [Security Notes](#security-notes)
|
||||
|
||||
**Detail lives in `docs/`:**
|
||||
|
||||
| Document | Covers |
|
||||
|----------|--------|
|
||||
| [docs/providers.md](docs/providers.md) | Session tokens and how to refresh them, the `auth` wizard, ChatGPT Projects, Claude Code sessions, Codex sessions |
|
||||
| [docs/cli.md](docs/cli.md) | CLI reference — every command and flag |
|
||||
| [docs/scheduling.md](docs/scheduling.md) | The daily run on Linux and Windows, and push notifications |
|
||||
| [docs/troubleshooting.md](docs/troubleshooting.md) | How the cache works, what each failure means, filing a bug report |
|
||||
|
||||
---
|
||||
|
||||
## Terms of Service Warning
|
||||
@@ -151,81 +154,6 @@ It probes one conversation per provider and checks only the fields the parser de
|
||||
|
||||
---
|
||||
|
||||
## Getting Your Session Tokens
|
||||
|
||||
Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
|
||||
|
||||
Tokens are entered manually — copy them from your browser's DevTools and run the wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
The wizard detects your OS, shows the correct DevTools shortcut, and writes the values to `.env` without echoing them to the terminal.
|
||||
|
||||
> **Why no auto-extraction from the browser?** Modern Chromium browsers (Chrome 127+, current Brave) encrypt cookies on Windows with **App-Bound Encryption**: the key is bound to the browser through a SYSTEM-level service, so reading cookies off disk requires Administrator rights and a PsExec-style SYSTEM impersonation that antivirus flags as credential theft — and it still fails on Brave specifically. There is no reliable, non-invasive way to pull these cookies automatically, so the manual DevTools flow below is the supported path.
|
||||
|
||||
### Token Lifetimes
|
||||
|
||||
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
||||
|----------|-------------|----------|-----------------|
|
||||
| ChatGPT | `__Secure-next-auth.session-token` (split into `.0` + `.1` when over ~4KB) | refresh ~weekly | `error` field of `/api/auth/session` — `doctor` reports "ChatGPT token active". The token is an encrypted JWE, so its `exp` is **not** readable client-side, and the `expires` field is a misleading rolling window; the `error` (`RefreshAccessTokenError` when dead) is the honest signal. |
|
||||
| Claude | `sessionKey` | ~30 days | Opaque token — only detectable from an API rejection. claude.ai answers an invalid session with **403** `permission_error` / `account_session_invalid`, **not** 401; `doctor` reports it on the "Claude API reachable" row. |
|
||||
|
||||
### Finding Tokens in Chrome DevTools
|
||||
|
||||
1. Open the provider's website and make sure you're logged in
|
||||
2. Press **F12** (Windows/Linux) or **Cmd+Option+I** (macOS) to open DevTools
|
||||
3. Click the **Application** tab
|
||||
4. In the left panel, expand **Cookies** and click the site URL
|
||||
5. Find the cookie by name and copy its **Value**
|
||||
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find the session token cookie. You will
|
||||
see **one of two layouts**, depending on how large your session token is:
|
||||
|
||||
- **One cookie**, `__Secure-next-auth.session-token` — copy Value → `CHATGPT_SESSION_TOKEN`, and leave `CHATGPT_SESSION_TOKEN_1` empty.
|
||||
- **Two cookies**, `__Secure-next-auth.session-token.0` and `.1` — copy `.0` (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`, and `.1` → `CHATGPT_SESSION_TOKEN_1`.
|
||||
|
||||
Chrome splits a cookie only when it exceeds ~4KB, so a larger session is chunked
|
||||
and a smaller one is not — the same account can differ from machine to machine.
|
||||
`CHATGPT_SESSION_TOKEN_1` is optional; both layouts authenticate, because the
|
||||
server reassembles a complete value sent under the `.0` name.
|
||||
|
||||
What does *not* work is sending a **partial** chunk — `.0` on its own when a `.1`
|
||||
exists. That fails silently: `/api/auth/session` answers HTTP 200 with no
|
||||
`accessToken` rather than an error. If you see two cookies, copy both.
|
||||
|
||||
**Claude:** go to `https://claude.ai` → find `sessionKey` → copy Value
|
||||
|
||||
### When Tokens Expire
|
||||
|
||||
An expired token shows up as an authentication error naming the cookie to
|
||||
refresh and how. The status differs by provider — ChatGPT reports 401, while
|
||||
claude.ai reports **403 "Invalid authorization"** (`account_session_invalid`) —
|
||||
so don't read a 403 from Claude as a permissions problem with your account.
|
||||
|
||||
To refresh:
|
||||
- Re-run the `auth` wizard: `ai-chat-exporter auth`
|
||||
- Or manually update the value in your `.env` file
|
||||
|
||||
`ai-chat-exporter doctor` is the quickest check: the "token set" rows only test
|
||||
that a value is present, so an expired credential passes those and fails on the
|
||||
"API reachable" row.
|
||||
|
||||
---
|
||||
|
||||
## The `auth` Command
|
||||
|
||||
The easiest way to configure tokens is the interactive wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
This walks you through finding your token, validates it, shows the expiry date (ChatGPT only), and offers to write it to your `.env` automatically. Tokens are never echoed to the terminal.
|
||||
|
||||
---
|
||||
|
||||
## `.env` Setup
|
||||
|
||||
Copy `.env.example` to `.env` and fill in your values:
|
||||
@@ -273,7 +201,7 @@ cp .env.example .env
|
||||
|
||||
A topic on public ntfy.sh is readable by anyone who knows its name, so
|
||||
notifications carry per-provider counts and a machine name only — never
|
||||
conversation titles. See [Getting notified](#getting-notified).
|
||||
conversation titles. See [Getting notified](docs/scheduling.md#getting-notified).
|
||||
|
||||
### Cache & logging
|
||||
|
||||
@@ -285,212 +213,6 @@ conversation titles. See [Getting notified](#getting-notified).
|
||||
|
||||
---
|
||||
|
||||
## ChatGPT Projects
|
||||
|
||||
ChatGPT project conversations are stored separately from your main conversation list and require extra configuration.
|
||||
|
||||
### Finding your project IDs
|
||||
|
||||
The quickest way is to let the exporter find them:
|
||||
|
||||
```
|
||||
ai-chat-exporter projects
|
||||
```
|
||||
|
||||
It lists every project your conversations actually belong to, marks which are
|
||||
missing from `.env`, and prints a paste-ready `CHATGPT_PROJECT_IDS=` line
|
||||
(`--write` updates `.env` for you). If your ChatGPT account does not include
|
||||
the project on conversation summaries, add `--deep` and it reads each
|
||||
conversation's detail instead — slower, one request per conversation, but
|
||||
complete.
|
||||
|
||||
Why it matters: project attribution is resolved from each conversation's own
|
||||
`gizmo_id`, so exports file correctly whether or not a project is configured.
|
||||
But the *listing* pass still needs `CHATGPT_PROJECT_IDS` — conversations that
|
||||
live only inside a project never appear in the default conversation list, so an
|
||||
unlisted project's chats are never fetched at all. Every export run also names
|
||||
any unconfigured project it encounters.
|
||||
|
||||
To find them by hand instead:
|
||||
|
||||
1. Open ChatGPT and click a Project in the left sidebar
|
||||
2. Look at the browser URL — it will look like:
|
||||
`https://chatgpt.com/g/g-p-68c2b2b3037c8191890036fb4ae3ed9f-my-project/project`
|
||||
3. Copy the `g-p-…` part (everything up to but not including the slug after the second `-`)
|
||||
|
||||
Add all your project IDs to `.env` as a comma-separated list:
|
||||
|
||||
```
|
||||
CHATGPT_PROJECT_IDS=g-p-68c2b2b3037c8191890036fb4ae3ed9f,g-p-anotherprojectid
|
||||
```
|
||||
|
||||
The `auth` wizard can also guide you through this step interactively.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Sessions
|
||||
|
||||
The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider claude-code
|
||||
ai-chat-exporter joplin --provider claude-code
|
||||
```
|
||||
|
||||
Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Sessions become notes under their own top-level **`AI-ClaudeCode`** Joplin notebook, in a sub-notebook per launch folder. The provider is included in `--provider all` whenever a sessions directory exists.
|
||||
|
||||
**Subagents.** Claude Code stores subagent (Task-tool) transcripts as separate files under `<session>/subagents/`; each is folded into its parent session inline, as a collapsible `<details>` block labeled with the subagent's type and description (its own tool traffic is collapsed like the main dialogue).
|
||||
|
||||
**Repo tags.** Sessions launched from a workspace root all share one folder-named notebook, so titles carry the repos each session touched — `Resume StartWRT project work [start-technologies]` — for at-a-glance scanning and search. A file's repo is the git repository it lives in (nearest ancestor with a `.git`), resolved from the tool paths in the transcript, so work is tagged wherever it happened — even across workspaces — and config/one-off files are ignored (they aren't repos). Repos are frequency-ordered and capped at 3. To never tag specific repos, set `CLAUDE_CODE_REPO_TAG_IGNORE` (comma-separated). Note this reads your current git layout, so a repo you later delete or move drops from the tag on re-export.
|
||||
|
||||
**Multiple locations.** By default the provider scans `~/.claude/projects/` plus `$CLAUDE_CONFIG_DIR/projects` when `CLAUDE_CONFIG_DIR` is set. To scan additional roots (e.g. other machines' sessions copied onto this box), set `CLAUDE_CODE_DIR` to a `:`-separated list:
|
||||
|
||||
```bash
|
||||
CLAUDE_CODE_DIR="$HOME/.claude/projects:/mnt/backup/laptop/.claude/projects"
|
||||
```
|
||||
|
||||
Sessions from all roots are merged by folder (no per-machine label); if the same session UUID appears in two roots, the newer copy wins. Note the exporter only sees this machine's disk and cannot recover sessions Claude Code has already pruned — run it regularly.
|
||||
|
||||
---
|
||||
|
||||
## Codex Sessions
|
||||
|
||||
The `codex` provider archives your local [Codex CLI](https://chatgpt.com/codex) agent transcripts — same deal as Claude Code: no tokens, no API, no ToS exposure. Rollout files are read from `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider codex
|
||||
ai-chat-exporter joplin --provider codex
|
||||
```
|
||||
|
||||
Exports are prose-only by default, with the same placeholder format as Claude Code, into their own top-level **`AI-Codex`** Joplin notebook. Repo tags work the same way (`CODEX_REPO_TAG_IGNORE` to suppress), and additional roots can be scanned with `CODEX_DIR` (`:`-separated); `CODEX_HOME`'s `sessions/` is picked up automatically when that variable is set.
|
||||
|
||||
Three things differ from Claude Code, all forced by how Codex stores its data:
|
||||
|
||||
**Reasoning cannot be exported.** Codex encrypts it at rest — every reasoning record carries `encrypted_content` with no plaintext summary in any layer of the file. It is always dropped and counted; `EXPORTER_HIDDEN_CONTENT=full` cannot bring it back.
|
||||
|
||||
**Incomplete tool calls are reported.** Codex writes each session twice in one file: its own typed items (what actually ran) and the raw model-facing wire format (everything attempted). The provider reads the typed layer — it is already decoded, and it omits harness plumbing that would otherwise need stripping — but cross-checks the raw layer for calls that never produced a result, so placeholders read `3 calls: exec_command ×3 (+2 did not complete)`. Those are commands that failed to launch, that you aborted, or that were still running when the turn ended.
|
||||
|
||||
**Cloud tasks are out of scope.** `codex cloud` tasks run server-side and are reachable at `chatgpt.com/backend-api/api/codex/tasks`, but local CLI sessions are never uploaded there, so the cloud API is not an alternative source for these transcripts and this provider stays entirely offline. If you start using `codex cloud exec`, those transcripts would be cloud-only and would need separate work.
|
||||
|
||||
---
|
||||
|
||||
## Scheduling a Daily Run
|
||||
|
||||
`sync` chains `export` then `joplin` in one invocation, which is what a scheduler
|
||||
wants — one command, and a meaningful exit code so a failed run is visible
|
||||
instead of silent.
|
||||
|
||||
```bash
|
||||
./ai-chat-exporter sync # every configured provider
|
||||
./ai-chat-exporter sync --provider codex # just one
|
||||
```
|
||||
|
||||
### Linux (systemd user timer)
|
||||
|
||||
```bash
|
||||
./scheduling/install-systemd-timer.sh --provider claude-code --provider codex
|
||||
./scheduling/install-systemd-timer.sh --uninstall
|
||||
```
|
||||
|
||||
Defaults to 09:00 daily (`--time 21:30` to change). `Persistent=true` means a
|
||||
machine that was off at the scheduled time runs the archive at next boot rather
|
||||
than skipping the day. To archive while logged out, `loginctl enable-linger $USER`.
|
||||
|
||||
Check on it with `systemctl --user list-timers aichat-sync.timer` and
|
||||
`journalctl --user -u aichat-sync.service -n 50`.
|
||||
|
||||
### Windows (Task Scheduler)
|
||||
|
||||
```powershell
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Provider chatgpt,claude
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Unregister
|
||||
```
|
||||
|
||||
Per-user task, no admin rights needed. `-StartWhenAvailable` is the counterpart
|
||||
of systemd's `Persistent=true`. The task runs `scheduling\run-sync.ps1` with its
|
||||
window hidden; Task Scheduler keeps no output, so each run's output is appended
|
||||
to `cache\logs\scheduled-sync.log` (rolled over to `.1` past 1 MB).
|
||||
|
||||
Check on it with `Get-ScheduledTaskInfo -TaskName AiChatExporterSync` (look at
|
||||
`LastTaskResult`: 0 is success) and
|
||||
`Get-Content cache\logs\scheduled-sync.log -Tail 50`.
|
||||
|
||||
### What to know before you rely on it
|
||||
|
||||
**List only the providers that work unattended on that machine.** The local
|
||||
providers (`claude-code`, `codex`) need no credentials and always work. The web
|
||||
providers depend on a session token that expires and can only be refreshed by
|
||||
hand via DevTools — so on a machine where that token is stale, scheduling them
|
||||
means a failed run every single day, which is a good way to learn to ignore
|
||||
failures you actually want to see. That is why `--provider` is repeatable in both
|
||||
installers: schedule the coding machine for `claude-code` + `codex`, the browser
|
||||
machine for `chatgpt` + `claude`.
|
||||
|
||||
**Both installers pass `--joplin-optional`.** If Joplin desktop isn't running,
|
||||
the sync warns instead of failing: the export has already captured the local
|
||||
transcripts (the part that can disappear), and the notes are rebuilt from the
|
||||
cache on the next run that finds Joplin up.
|
||||
|
||||
**One provider failing does not skip the rest.** Both installers loop over the
|
||||
providers in a single action rather than one action each, because systemd
|
||||
`oneshot` stops at the first failing `ExecStart` and Task Scheduler reports only
|
||||
the last action's result. Every provider is attempted; the run still exits
|
||||
non-zero if any failed. The loop is `scheduling/run-sync.sh` on Linux (the
|
||||
unit's `ExecStart`) and `scheduling\run-sync.ps1` on Windows (the task's action).
|
||||
|
||||
### Getting notified
|
||||
|
||||
A scheduled run is silent by default. Output goes to three pull-only places: the
|
||||
exporter's own log (`cache/logs/exporter.log`), the systemd journal on Linux
|
||||
(`journalctl --user -u aichat-sync.service`), and on Windows Task Scheduler's
|
||||
`LastTaskResult` plus `cache\logs\scheduled-sync.log`.
|
||||
|
||||
To have runs report back, set an [ntfy](https://ntfy.sh) topic in `.env`:
|
||||
|
||||
```bash
|
||||
NTFY_TOPIC=my-archive-topic
|
||||
```
|
||||
|
||||
Then check it works before waiting on a scheduled run:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter notify # show settings
|
||||
ai-chat-exporter notify --test # send a test push
|
||||
```
|
||||
|
||||
`sync` pushes a result whenever a topic is configured — a success carries the
|
||||
per-provider counts at low/default priority, a failure carries the reason at
|
||||
high priority with an alert tag, so a failed archive is distinguishable from a
|
||||
quiet one on your phone. `NTFY_NOTIFY=failure` notifies only on failure; `off`
|
||||
disables it; `--notify` / `--no-notify` override per run.
|
||||
|
||||
`sync` can only push from the end of a run it finished. A crash, an exit before
|
||||
the sync starts (the ToS gate, a cache error) or a launcher that can't build its
|
||||
venv sends nothing — and since each provider pushes separately, the ones that
|
||||
succeeded still say "OK", so a dead provider looks like a quiet day. The
|
||||
scheduler wrappers (`scheduling/run-sync.sh`, `scheduling\run-sync.ps1`) close
|
||||
that gap: any run that exits non-zero without the
|
||||
app having reported it gets a high-priority **FAILED** push naming the provider
|
||||
and, for a crash, the exception's class (`claude-code: crashed (RecursionError)`)
|
||||
— the class only, never its message, which can carry a conversation title. The
|
||||
traceback is in the journal on Linux and in `cache\logs\scheduled-sync.log` on
|
||||
Windows.
|
||||
|
||||
The message includes the **machine name**, which matters because both machines
|
||||
archive into one topic. It contains counts only — never conversation titles. A
|
||||
topic on public ntfy.sh is readable by anyone who knows its name, so if you want
|
||||
it private, self-host (`NTFY_SERVER`) or use an access-controlled topic with
|
||||
`NTFY_TOKEN`.
|
||||
|
||||
A notification is never fatal: if ntfy is unreachable, the run logs a warning and
|
||||
still reports its real exit code.
|
||||
|
||||
**Acknowledge the ToS notice once, interactively.** It's stored in the cache
|
||||
manifest per machine. Until then a scheduled run exits 1 with an explanation
|
||||
rather than hanging on a prompt no one can answer.
|
||||
|
||||
---
|
||||
|
||||
## Output Structure
|
||||
|
||||
All exported files go under `EXPORT_DIR`. The folder structure maps directly to Joplin notebooks.
|
||||
@@ -540,392 +262,34 @@ Each provider+project combination maps to a flat Joplin notebook created automat
|
||||
|
||||
---
|
||||
|
||||
## CLI Reference
|
||||
|
||||
### Global flags
|
||||
|
||||
```
|
||||
--verbose / -v DEBUG output to console
|
||||
--quiet / -q WARNING and above only
|
||||
--debug DEBUG + full tracebacks + redacted API response bodies
|
||||
--no-log-file Disable file logging
|
||||
--version Print version and exit
|
||||
```
|
||||
|
||||
### `auth` — Interactive token setup
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth # manual wizard (DevTools flow)
|
||||
```
|
||||
|
||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut, then writes the values to `.env`.
|
||||
|
||||
### `doctor` — Health check
|
||||
|
||||
```bash
|
||||
ai-chat-exporter doctor
|
||||
```
|
||||
|
||||
Checks: token presence, JWT validity and expiry, directory permissions, disk space, live API reachability. Exits with code 0 if all pass, 1 if any fail.
|
||||
|
||||
### `export` — Export conversations
|
||||
|
||||
```bash
|
||||
# Export everything (new/updated only)
|
||||
ai-chat-exporter export
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter export --provider claude
|
||||
|
||||
# JSON output
|
||||
ai-chat-exporter export --format json
|
||||
|
||||
# Both Markdown and JSON
|
||||
ai-chat-exporter export --format both
|
||||
|
||||
# Only conversations updated since a date
|
||||
ai-chat-exporter export --since 2024-06-01
|
||||
|
||||
# Only conversations in a specific project (case-insensitive substring)
|
||||
ai-chat-exporter export --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter export --project none
|
||||
|
||||
# Write to a custom directory
|
||||
ai-chat-exporter export --output /path/to/my/notes
|
||||
|
||||
# Preview without writing anything
|
||||
ai-chat-exporter export --dry-run
|
||||
```
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--force`, `--dry-run`
|
||||
|
||||
**Re-rendering the whole archive after an upgrade.** New formatting or features (collapse policy, media downloads) only change conversations as they're re-exported. To re-render everything you already have, use `--force` — it re-exports every conversation even if unchanged, **without** `cache --clear`, so your Joplin note links are preserved (a later `joplin` run updates the existing notes instead of duplicating them).
|
||||
|
||||
A force re-render runs as a tracked "campaign": each run re-renders the least-recently-exported conversations, the "still to go" count shrinks each run, and finished providers do no further work. The simplest approach is one uncapped pass:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force # re-renders everything in one paced pass
|
||||
ai-chat-exporter joplin # update notes + upload media
|
||||
```
|
||||
|
||||
Or spread the load with `--max-conversations`, re-running until it reports "Force re-render complete":
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force --max-conversations 50 # repeat until complete
|
||||
```
|
||||
|
||||
### `list` — List conversations
|
||||
|
||||
```bash
|
||||
# List all conversations for all providers
|
||||
ai-chat-exporter list
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter list --provider chatgpt
|
||||
|
||||
# Filter by project
|
||||
ai-chat-exporter list --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter list --project none
|
||||
```
|
||||
|
||||
Fetches and displays all conversations without exporting them. Useful for verifying what the tool can see before running an export.
|
||||
|
||||
### `joplin` — Sync to Joplin
|
||||
|
||||
```bash
|
||||
# Sync all pending conversations to Joplin
|
||||
ai-chat-exporter joplin
|
||||
|
||||
# Preview what would be synced without sending anything
|
||||
ai-chat-exporter joplin --dry-run
|
||||
|
||||
# Sync a single provider
|
||||
ai-chat-exporter joplin --provider chatgpt
|
||||
|
||||
# Sync only conversations in a specific project
|
||||
ai-chat-exporter joplin --project "learning python"
|
||||
|
||||
# Sync only conversations outside any project
|
||||
ai-chat-exporter joplin --project none
|
||||
```
|
||||
|
||||
Reads the local export cache and pushes each exported Markdown file to Joplin as a note. Notebooks are created automatically. Re-running is safe — notes are updated (not duplicated).
|
||||
|
||||
**Prerequisites:**
|
||||
1. Run `export` first to generate the Markdown files
|
||||
2. Open Joplin → Tools → Options → Web Clipper → enable the service
|
||||
3. Copy the Authorization token and add `JOPLIN_API_TOKEN=<token>` to your `.env`
|
||||
4. Joplin desktop must be open when you run this command
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--project NAME`, `--dry-run`
|
||||
|
||||
### `sync` — Export and sync in one run
|
||||
|
||||
```bash
|
||||
# The whole archive run: export, then push to Joplin
|
||||
ai-chat-exporter sync
|
||||
|
||||
# One provider
|
||||
ai-chat-exporter sync --provider codex
|
||||
|
||||
# Export only; don't touch Joplin
|
||||
ai-chat-exporter sync --skip-joplin
|
||||
|
||||
# Joplin being closed is a warning, not a failure (used by the schedulers)
|
||||
ai-chat-exporter sync --joplin-optional
|
||||
```
|
||||
|
||||
Equivalent to `export` followed by `joplin` with the same `--provider`. Intended
|
||||
for scheduled runs — see [Scheduling a Daily Run](#scheduling-a-daily-run).
|
||||
|
||||
Unlike the individual commands, `sync` sets a **meaningful exit code**: non-zero
|
||||
if any conversation failed to export or any note failed to sync. A provider whose
|
||||
listing call fails outright (an expired web session token being the usual cause)
|
||||
counts its whole batch as failed. A provider that is simply unconfigured, or that
|
||||
had nothing new, is ordinary success. `export` on its own always exits 0, which
|
||||
is fine when you're reading the summary table and useless to a scheduler.
|
||||
|
||||
`--joplin-optional` downgrades an unreachable Joplin to a warning: the export has
|
||||
already captured the local transcripts, and the notes are rebuilt from the cache
|
||||
by the next run that finds Joplin open.
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--since YYYY-MM-DD`, `--hidden-content [full|placeholder|omit]`, `--max-conversations N`, `--skip-joplin`, `--joplin-optional`, `--notify/--no-notify`, `--dry-run`
|
||||
|
||||
Note this is a deliberate subset of `export`'s options — `--format`, `--output`,
|
||||
`--project`, `--download-media` and `--force` are not passed through. Use
|
||||
`export` directly for those. (`--download-media` still applies from `.env`; the
|
||||
flag is only a per-run override.)
|
||||
|
||||
### `notify` — Push-notification settings and test
|
||||
|
||||
```bash
|
||||
# Show the current settings
|
||||
ai-chat-exporter notify
|
||||
|
||||
# Send a test push to confirm the topic works
|
||||
ai-chat-exporter notify --test
|
||||
```
|
||||
|
||||
Shows the resolved ntfy configuration and which machine name will appear in the
|
||||
title. See [Getting notified](#getting-notified) for what a scheduled run sends.
|
||||
|
||||
Options: `--test`
|
||||
|
||||
### `prune` — Delete stale export files
|
||||
|
||||
```bash
|
||||
# Preview what would be deleted
|
||||
ai-chat-exporter prune --dry-run
|
||||
|
||||
# Delete (asks for confirmation; -y skips the prompt)
|
||||
ai-chat-exporter prune
|
||||
```
|
||||
|
||||
Deletes export files no longer referenced by the cache manifest — leftovers
|
||||
from old folder layouts or fixed bugs that would otherwise double-sync into
|
||||
Joplin. Refuses to run when the manifest is empty (e.g. right after
|
||||
`cache --clear`) so it can never wipe a freshly cleared archive. The `doctor`
|
||||
command separately verifies that every manifest entry's file exists on disk.
|
||||
|
||||
### `projects` — Discover ChatGPT project IDs
|
||||
|
||||
```bash
|
||||
# List the projects your conversations belong to
|
||||
ai-chat-exporter projects
|
||||
|
||||
# Also inspect conversations whose listing entry doesn't name a project
|
||||
ai-chat-exporter projects --deep
|
||||
|
||||
# Write the discovered IDs straight into .env
|
||||
ai-chat-exporter projects --write
|
||||
```
|
||||
|
||||
`CHATGPT_PROJECT_IDS` is maintained by hand, and a project missing from it is
|
||||
invisible to the listing pass — conversations that live *only* inside that
|
||||
project are never fetched at all. This reports every project your conversations
|
||||
belong to, marks the ones absent from `.env`, and prints a paste-ready line.
|
||||
|
||||
`--deep` fetches each conversation's detail when the listing doesn't name its
|
||||
project: complete, but one request per conversation, so it's slow.
|
||||
|
||||
Options: `--deep`, `--write`
|
||||
|
||||
### `canary` — Check for provider API drift
|
||||
|
||||
```bash
|
||||
ai-chat-exporter canary
|
||||
ai-chat-exporter canary --provider chatgpt
|
||||
```
|
||||
|
||||
The web providers are undocumented internal APIs that can change shape without
|
||||
notice, and the failure mode is silent — a renamed field means content is
|
||||
quietly dropped rather than an error being raised. The canary fetches one
|
||||
listing page and one conversation per provider and asserts only the fields the
|
||||
normalizer actually depends on.
|
||||
|
||||
Findings are `ERROR` (a load-bearing field is missing or mistyped — the parser
|
||||
will break or silently lose data) or `WARN` (something unfamiliar appeared;
|
||||
worth investigating, not necessarily broken). **Exits non-zero on any ERROR**,
|
||||
so it can be scheduled or run in CI. Local providers have no remote schema and
|
||||
are not probed.
|
||||
|
||||
Options: `--provider [chatgpt|claude|all]`
|
||||
|
||||
### `cache` — Manage the sync manifest
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
ai-chat-exporter cache --show
|
||||
|
||||
# Clear all cached entries (forces full re-export next run)
|
||||
ai-chat-exporter cache --clear
|
||||
|
||||
# Clear a single provider
|
||||
ai-chat-exporter cache --clear --provider claude
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How the Cache Works
|
||||
|
||||
The cache manifest lives at `cache/manifest.json` (inside the install directory) and records every exported conversation: its title, project, `updated_at` timestamp, output file path, and (after Joplin sync) the Joplin note ID.
|
||||
|
||||
On every `export` run:
|
||||
1. Fetch the full conversation list from the provider
|
||||
2. Compare each conversation's `updated_at` against the manifest
|
||||
3. Export only conversations that are new or have been updated
|
||||
4. Write each successfully exported conversation to the manifest **immediately** (not batched)
|
||||
|
||||
On every `joplin` run:
|
||||
1. Read the manifest to find conversations not yet synced to Joplin, or re-exported since last sync
|
||||
2. Push each pending Markdown file to Joplin (create or update)
|
||||
3. Store the Joplin note ID in the manifest so subsequent runs update rather than duplicate
|
||||
|
||||
**This design makes every run inherently resumable.** If the tool is interrupted for any reason — rate limit, network drop, Ctrl+C, crash — simply re-run the same command. It will skip already-processed conversations and continue from where it stopped.
|
||||
|
||||
A conversation whose assets failed transiently also carries `pending_media` (the
|
||||
asset refs still missing) and `pending_media_attempts`. While `pending_media` is
|
||||
set and the attempt cap is not reached, step 2 treats the conversation as stale
|
||||
so the next run re-fetches those assets — see
|
||||
[Embedded images and media](#embedded-images-and-media). A run that recovers them
|
||||
clears the key.
|
||||
|
||||
To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
|
||||
|
||||
To re-export a single conversation, delete its entry from `cache/manifest.json`
|
||||
and re-run. (There is deliberately no `cache --reset --conversation` flag; see
|
||||
`FUTURE.md`.)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `Authentication failed` (401, or 403 from Claude)
|
||||
Your session token has expired.
|
||||
- Run `ai-chat-exporter auth` to get a new token interactively
|
||||
- Or manually copy a fresh cookie value into your `.env` file
|
||||
|
||||
Note: neither token's expiry can be read client-side. Claude's `sessionKey` is an opaque string, and claude.ai reports an invalid one as **403** "Invalid authorization" (`account_session_invalid`), not 401. ChatGPT's token is an encrypted JWE; `doctor` reads the `error` field of `/api/auth/session` instead. See [When Tokens Expire](#when-tokens-expire).
|
||||
|
||||
### `429 Rate Limited`
|
||||
The tool automatically pauses, saves progress, and exits with a clear message showing how many conversations were exported vs remaining. Just re-run the same export command to resume — the cache picks up exactly where it left off.
|
||||
|
||||
### `HTTP 500` / `503` on an asset download
|
||||
The provider's asset storage, not your setup. Retries and the cross-run recovery
|
||||
described in [Embedded images and media](#embedded-images-and-media) handle this;
|
||||
the run reports `media failed` and the next run re-fetches. Measured 2026-10-20:
|
||||
two assets lost this way to a 500 and a 503 both downloaded cleanly an hour later.
|
||||
|
||||
### Joplin: "JOPLIN_API_TOKEN is not set"
|
||||
You need to configure the token before running the `joplin` command:
|
||||
1. Open Joplin desktop
|
||||
2. Go to Tools → Options → Web Clipper
|
||||
3. Enable the Web Clipper service
|
||||
4. Copy the Authorization token shown on that page
|
||||
5. Add `JOPLIN_API_TOKEN=<token>` to your `.env` file
|
||||
|
||||
### Joplin: "Joplin is not responding"
|
||||
Joplin desktop must be running when you run the `joplin` command. The Web Clipper service shuts down when Joplin is closed.
|
||||
|
||||
### Joplin: "Joplin rejected the API token (HTTP 401)"
|
||||
The token in `JOPLIN_API_TOKEN` doesn't match what Joplin expects. Get a fresh token from Joplin → Tools → Options → Web Clipper → Authorization token.
|
||||
|
||||
### Joplin: note timed out
|
||||
If you see a timeout error, Joplin took longer than `JOPLIN_REQUEST_TIMEOUT` seconds (default: 30) to respond. Possible causes:
|
||||
- The conversation is very large and Joplin is slow to index it
|
||||
- Joplin is busy syncing or loading a large library
|
||||
- Joplin has frozen — try restarting it
|
||||
|
||||
To increase the timeout: add `JOPLIN_REQUEST_TIMEOUT=60` to your `.env`.
|
||||
|
||||
### ChatGPT project conversations not appearing
|
||||
Make sure you've added the project IDs to `CHATGPT_PROJECT_IDS` in your `.env`. See [ChatGPT Projects](#chatgpt-projects) for how to find them. Project conversations are not included in the default conversation listing — they must be fetched separately.
|
||||
|
||||
### Schema warnings in logs (`Unexpected API response shape`)
|
||||
The provider's internal API may have changed. Run with `--debug`, sanitize the output (remove any personal content), and check the project's GitHub Issues for known fixes.
|
||||
|
||||
### Non-text content warnings
|
||||
Since v0.4.0, rich content is preserved as typed blocks in the export. ChatGPT voice transcripts render as text and audio assets as `📎 File attached` placeholders with size and duration metadata. Anything the extractor doesn't recognise renders as a visible `> ⚠️ Unsupported content` block naming the type and observed keys, *and* increments a counter in the post-export summary so you can tell whether real content is being silently skipped.
|
||||
|
||||
### Embedded images and media
|
||||
Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as ``. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
|
||||
|
||||
A failed asset **never fails the export** — the conversation is archived with its
|
||||
text placeholder, and the failure is tallied as `media failed` in the run summary
|
||||
and in the push notification. What happens next depends on *why* it failed:
|
||||
|
||||
| Bucket | Meaning | Retried? |
|
||||
|---|---|---|
|
||||
| `expired-or-missing` | 404 — the upload was deleted server-side | No. The bytes are gone. |
|
||||
| `forbidden` | 403 — metadata survives, bytes do not (see the 2026-08-17 investigation in `CHANGELOG.md`) | No. |
|
||||
| `download-error` | 5xx, a timeout, a network blip | **Yes** — within the run, then on up to 3 later runs |
|
||||
| `skipped-provider-unwell` | the run's circuit breaker was open | **Yes**, on a later run |
|
||||
| `unparseable-ref` / `write-error` | local problem, not the provider's | No |
|
||||
|
||||
**Within a run**, an asset download is two requests — one for a signed URL, one
|
||||
for the bytes — and each is retried on a 5xx: immediately once, then after about
|
||||
2, 4, 8 and 16 seconds with jitter, honouring `Retry-After` when the server sends
|
||||
one. Retries for a single asset are capped by a wall-clock budget so a hanging
|
||||
endpoint cannot consume the run's whole execution window.
|
||||
|
||||
**Across runs**, an asset that failed transiently is recorded against its
|
||||
conversation in the cache manifest, and the next run re-exports that conversation
|
||||
to re-fetch it. Only the missing asset is fetched — media already on disk is
|
||||
reused, and Joplin resource IDs are keyed by path, so the re-sync uploads just
|
||||
the new file. After 3 unsuccessful runs the asset is treated as lost and the
|
||||
conversation stops being re-exported.
|
||||
|
||||
**Circuit breaker.** If 5 transient failures land back to back, asset downloads
|
||||
stop for the rest of the run and the remaining assets are recorded for a later
|
||||
attempt. A provider's storage having a bad day should cost one quiet morning, not
|
||||
a run that overruns its two-hour scheduled window. Permanent failures never count
|
||||
toward this — a deleted upload says nothing about the provider's health.
|
||||
|
||||
### Tool output / hidden context collapsed
|
||||
Since v0.6.0, content that was invisible in the ChatGPT web UI is collapsed by default to one-line placeholders like `> 🔧 Tool output — file_search (15.0 KB) — omitted`. This is ChatGPT's file-retrieval tool re-injecting your attached files on every run — it can be 90% of a conversation's bytes while containing none of the dialogue. Custom Instructions collapse the same way (`> ℹ️ Hidden context`). The post-export summary lists every collapsed message by origin with the total KB omitted. To keep everything, set `EXPORTER_HIDDEN_CONTENT=full` in `.env` or pass `--hidden-content full`.
|
||||
|
||||
### Empty export / all conversations skipped
|
||||
No new or updated conversations since your last run. To verify: `ai-chat-exporter cache --show`. To force a full re-export: `ai-chat-exporter cache --clear`.
|
||||
|
||||
### Filing a bug report
|
||||
1. Run with `--debug`: `ai-chat-exporter export --debug 2>&1 | tee debug.log`
|
||||
2. Remove any personal conversation content from `debug.log`
|
||||
3. Open a GitHub Issue with the sanitized log and the exact command you ran
|
||||
## Documentation
|
||||
|
||||
This page covers installing and configuring the exporter. Everything else is in
|
||||
`docs/`:
|
||||
|
||||
- **[Providers and Tokens](docs/providers.md)** — getting a session token out of
|
||||
DevTools, what each one's lifetime is and how expiry is detected, the `auth`
|
||||
wizard, and the setup each provider needs: ChatGPT Projects (which are not in
|
||||
the default conversation listing), Claude Code sessions, Codex sessions.
|
||||
- **[CLI Reference](docs/cli.md)** — `auth`, `doctor`, `export`, `list`,
|
||||
`joplin`, `sync`, `notify`, `prune`, `projects`, `canary`, `cache`, with the
|
||||
flags and examples for each.
|
||||
- **[Running on a Schedule](docs/scheduling.md)** — the systemd user timer, the
|
||||
Windows scheduled task, what to know before relying on either, and how run
|
||||
results reach your phone.
|
||||
- **[Troubleshooting](docs/troubleshooting.md)** — how the cache decides what to
|
||||
re-export, every error message the tool produces and what it means, media
|
||||
download failures, and what to include in a bug report.
|
||||
|
||||
---
|
||||
|
||||
## Future Work
|
||||
|
||||
See `FUTURE.md` for the full roadmap. Current priorities:
|
||||
See `FUTURE.md` for the full roadmap. Current priority:
|
||||
|
||||
- **A StartOS service** that centralises every machine's conversations into one
|
||||
corpus and owns the Joplin connection, so each machine only has to upload
|
||||
(`FUTURE.md` §8)
|
||||
- **Splitting this README** into a short overview plus separate documents
|
||||
(`FUTURE.md` §1)
|
||||
|
||||
---
|
||||
|
||||
|
||||
+254
@@ -0,0 +1,254 @@
|
||||
# CLI Reference
|
||||
|
||||
Every command, its flags and what it does.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [Scheduling](scheduling.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Global flags
|
||||
|
||||
```
|
||||
--verbose / -v DEBUG output to console
|
||||
--quiet / -q WARNING and above only
|
||||
--debug DEBUG + full tracebacks + redacted API response bodies
|
||||
--no-log-file Disable file logging
|
||||
--version Print version and exit
|
||||
```
|
||||
|
||||
## `auth` — Interactive token setup
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth # manual wizard (DevTools flow)
|
||||
```
|
||||
|
||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut, then writes the values to `.env`.
|
||||
|
||||
## `doctor` — Health check
|
||||
|
||||
```bash
|
||||
ai-chat-exporter doctor
|
||||
```
|
||||
|
||||
Checks: token presence, JWT validity and expiry, directory permissions, disk space, live API reachability. Exits with code 0 if all pass, 1 if any fail.
|
||||
|
||||
## `export` — Export conversations
|
||||
|
||||
```bash
|
||||
# Export everything (new/updated only)
|
||||
ai-chat-exporter export
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter export --provider claude
|
||||
|
||||
# JSON output
|
||||
ai-chat-exporter export --format json
|
||||
|
||||
# Both Markdown and JSON
|
||||
ai-chat-exporter export --format both
|
||||
|
||||
# Only conversations updated since a date
|
||||
ai-chat-exporter export --since 2024-06-01
|
||||
|
||||
# Only conversations in a specific project (case-insensitive substring)
|
||||
ai-chat-exporter export --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter export --project none
|
||||
|
||||
# Write to a custom directory
|
||||
ai-chat-exporter export --output /path/to/my/notes
|
||||
|
||||
# Preview without writing anything
|
||||
ai-chat-exporter export --dry-run
|
||||
```
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--force`, `--dry-run`
|
||||
|
||||
**Re-rendering the whole archive after an upgrade.** New formatting or features (collapse policy, media downloads) only change conversations as they're re-exported. To re-render everything you already have, use `--force` — it re-exports every conversation even if unchanged, **without** `cache --clear`, so your Joplin note links are preserved (a later `joplin` run updates the existing notes instead of duplicating them).
|
||||
|
||||
A force re-render runs as a tracked "campaign": each run re-renders the least-recently-exported conversations, the "still to go" count shrinks each run, and finished providers do no further work. The simplest approach is one uncapped pass:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force # re-renders everything in one paced pass
|
||||
ai-chat-exporter joplin # update notes + upload media
|
||||
```
|
||||
|
||||
Or spread the load with `--max-conversations`, re-running until it reports "Force re-render complete":
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force --max-conversations 50 # repeat until complete
|
||||
```
|
||||
|
||||
## `list` — List conversations
|
||||
|
||||
```bash
|
||||
# List all conversations for all providers
|
||||
ai-chat-exporter list
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter list --provider chatgpt
|
||||
|
||||
# Filter by project
|
||||
ai-chat-exporter list --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter list --project none
|
||||
```
|
||||
|
||||
Fetches and displays all conversations without exporting them. Useful for verifying what the tool can see before running an export.
|
||||
|
||||
## `joplin` — Sync to Joplin
|
||||
|
||||
```bash
|
||||
# Sync all pending conversations to Joplin
|
||||
ai-chat-exporter joplin
|
||||
|
||||
# Preview what would be synced without sending anything
|
||||
ai-chat-exporter joplin --dry-run
|
||||
|
||||
# Sync a single provider
|
||||
ai-chat-exporter joplin --provider chatgpt
|
||||
|
||||
# Sync only conversations in a specific project
|
||||
ai-chat-exporter joplin --project "learning python"
|
||||
|
||||
# Sync only conversations outside any project
|
||||
ai-chat-exporter joplin --project none
|
||||
```
|
||||
|
||||
Reads the local export cache and pushes each exported Markdown file to Joplin as a note. Notebooks are created automatically. Re-running is safe — notes are updated (not duplicated).
|
||||
|
||||
**Prerequisites:**
|
||||
1. Run `export` first to generate the Markdown files
|
||||
2. Open Joplin → Tools → Options → Web Clipper → enable the service
|
||||
3. Copy the Authorization token and add `JOPLIN_API_TOKEN=<token>` to your `.env`
|
||||
4. Joplin desktop must be open when you run this command
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--project NAME`, `--dry-run`
|
||||
|
||||
## `sync` — Export and sync in one run
|
||||
|
||||
```bash
|
||||
# The whole archive run: export, then push to Joplin
|
||||
ai-chat-exporter sync
|
||||
|
||||
# One provider
|
||||
ai-chat-exporter sync --provider codex
|
||||
|
||||
# Export only; don't touch Joplin
|
||||
ai-chat-exporter sync --skip-joplin
|
||||
|
||||
# Joplin being closed is a warning, not a failure (used by the schedulers)
|
||||
ai-chat-exporter sync --joplin-optional
|
||||
```
|
||||
|
||||
Equivalent to `export` followed by `joplin` with the same `--provider`. Intended
|
||||
for scheduled runs — see [Scheduling a Daily Run](scheduling.md#scheduling-a-daily-run).
|
||||
|
||||
Unlike the individual commands, `sync` sets a **meaningful exit code**: non-zero
|
||||
if any conversation failed to export or any note failed to sync. A provider whose
|
||||
listing call fails outright (an expired web session token being the usual cause)
|
||||
counts its whole batch as failed. A provider that is simply unconfigured, or that
|
||||
had nothing new, is ordinary success. `export` on its own always exits 0, which
|
||||
is fine when you're reading the summary table and useless to a scheduler.
|
||||
|
||||
`--joplin-optional` downgrades an unreachable Joplin to a warning: the export has
|
||||
already captured the local transcripts, and the notes are rebuilt from the cache
|
||||
by the next run that finds Joplin open.
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--since YYYY-MM-DD`, `--hidden-content [full|placeholder|omit]`, `--max-conversations N`, `--skip-joplin`, `--joplin-optional`, `--notify/--no-notify`, `--dry-run`
|
||||
|
||||
Note this is a deliberate subset of `export`'s options — `--format`, `--output`,
|
||||
`--project`, `--download-media` and `--force` are not passed through. Use
|
||||
`export` directly for those. (`--download-media` still applies from `.env`; the
|
||||
flag is only a per-run override.)
|
||||
|
||||
## `notify` — Push-notification settings and test
|
||||
|
||||
```bash
|
||||
# Show the current settings
|
||||
ai-chat-exporter notify
|
||||
|
||||
# Send a test push to confirm the topic works
|
||||
ai-chat-exporter notify --test
|
||||
```
|
||||
|
||||
Shows the resolved ntfy configuration and which machine name will appear in the
|
||||
title. See [Getting notified](scheduling.md#getting-notified) for what a scheduled run sends.
|
||||
|
||||
Options: `--test`
|
||||
|
||||
## `prune` — Delete stale export files
|
||||
|
||||
```bash
|
||||
# Preview what would be deleted
|
||||
ai-chat-exporter prune --dry-run
|
||||
|
||||
# Delete (asks for confirmation; -y skips the prompt)
|
||||
ai-chat-exporter prune
|
||||
```
|
||||
|
||||
Deletes export files no longer referenced by the cache manifest — leftovers
|
||||
from old folder layouts or fixed bugs that would otherwise double-sync into
|
||||
Joplin. Refuses to run when the manifest is empty (e.g. right after
|
||||
`cache --clear`) so it can never wipe a freshly cleared archive. The `doctor`
|
||||
command separately verifies that every manifest entry's file exists on disk.
|
||||
|
||||
## `projects` — Discover ChatGPT project IDs
|
||||
|
||||
```bash
|
||||
# List the projects your conversations belong to
|
||||
ai-chat-exporter projects
|
||||
|
||||
# Also inspect conversations whose listing entry doesn't name a project
|
||||
ai-chat-exporter projects --deep
|
||||
|
||||
# Write the discovered IDs straight into .env
|
||||
ai-chat-exporter projects --write
|
||||
```
|
||||
|
||||
`CHATGPT_PROJECT_IDS` is maintained by hand, and a project missing from it is
|
||||
invisible to the listing pass — conversations that live *only* inside that
|
||||
project are never fetched at all. This reports every project your conversations
|
||||
belong to, marks the ones absent from `.env`, and prints a paste-ready line.
|
||||
|
||||
`--deep` fetches each conversation's detail when the listing doesn't name its
|
||||
project: complete, but one request per conversation, so it's slow.
|
||||
|
||||
Options: `--deep`, `--write`
|
||||
|
||||
## `canary` — Check for provider API drift
|
||||
|
||||
```bash
|
||||
ai-chat-exporter canary
|
||||
ai-chat-exporter canary --provider chatgpt
|
||||
```
|
||||
|
||||
The web providers are undocumented internal APIs that can change shape without
|
||||
notice, and the failure mode is silent — a renamed field means content is
|
||||
quietly dropped rather than an error being raised. The canary fetches one
|
||||
listing page and one conversation per provider and asserts only the fields the
|
||||
normalizer actually depends on.
|
||||
|
||||
Findings are `ERROR` (a load-bearing field is missing or mistyped — the parser
|
||||
will break or silently lose data) or `WARN` (something unfamiliar appeared;
|
||||
worth investigating, not necessarily broken). **Exits non-zero on any ERROR**,
|
||||
so it can be scheduled or run in CI. Local providers have no remote schema and
|
||||
are not probed.
|
||||
|
||||
Options: `--provider [chatgpt|claude|all]`
|
||||
|
||||
## `cache` — Manage the sync manifest
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
ai-chat-exporter cache --show
|
||||
|
||||
# Clear all cached entries (forces full re-export next run)
|
||||
ai-chat-exporter cache --clear
|
||||
|
||||
# Clear a single provider
|
||||
ai-chat-exporter cache --clear --provider claude
|
||||
```
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
# Providers and Tokens
|
||||
|
||||
How to obtain and refresh the session tokens for the two web providers, and the
|
||||
per-provider setup for ChatGPT Projects, Claude Code and Codex.
|
||||
|
||||
← [README](../README.md) · [Configuration](../README.md#env-setup) · [CLI Reference](cli.md) · [Scheduling](scheduling.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Getting Your Session Tokens
|
||||
|
||||
Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
|
||||
|
||||
Tokens are entered manually — copy them from your browser's DevTools and run the wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
The wizard detects your OS, shows the correct DevTools shortcut, and writes the values to `.env` without echoing them to the terminal.
|
||||
|
||||
> **Why no auto-extraction from the browser?** Modern Chromium browsers (Chrome 127+, current Brave) encrypt cookies on Windows with **App-Bound Encryption**: the key is bound to the browser through a SYSTEM-level service, so reading cookies off disk requires Administrator rights and a PsExec-style SYSTEM impersonation that antivirus flags as credential theft — and it still fails on Brave specifically. There is no reliable, non-invasive way to pull these cookies automatically, so the manual DevTools flow below is the supported path.
|
||||
|
||||
### Token Lifetimes
|
||||
|
||||
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
||||
|----------|-------------|----------|-----------------|
|
||||
| ChatGPT | `__Secure-next-auth.session-token` (split into `.0` + `.1` when over ~4KB) | refresh ~weekly | `error` field of `/api/auth/session` — `doctor` reports "ChatGPT token active". The token is an encrypted JWE, so its `exp` is **not** readable client-side, and the `expires` field is a misleading rolling window; the `error` (`RefreshAccessTokenError` when dead) is the honest signal. |
|
||||
| Claude | `sessionKey` | ~30 days | Opaque token — only detectable from an API rejection. claude.ai answers an invalid session with **403** `permission_error` / `account_session_invalid`, **not** 401; `doctor` reports it on the "Claude API reachable" row. |
|
||||
|
||||
### Finding Tokens in Chrome DevTools
|
||||
|
||||
1. Open the provider's website and make sure you're logged in
|
||||
2. Press **F12** (Windows/Linux) or **Cmd+Option+I** (macOS) to open DevTools
|
||||
3. Click the **Application** tab
|
||||
4. In the left panel, expand **Cookies** and click the site URL
|
||||
5. Find the cookie by name and copy its **Value**
|
||||
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find the session token cookie. You will
|
||||
see **one of two layouts**, depending on how large your session token is:
|
||||
|
||||
- **One cookie**, `__Secure-next-auth.session-token` — copy Value → `CHATGPT_SESSION_TOKEN`, and leave `CHATGPT_SESSION_TOKEN_1` empty.
|
||||
- **Two cookies**, `__Secure-next-auth.session-token.0` and `.1` — copy `.0` (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`, and `.1` → `CHATGPT_SESSION_TOKEN_1`.
|
||||
|
||||
Chrome splits a cookie only when it exceeds ~4KB, so a larger session is chunked
|
||||
and a smaller one is not — the same account can differ from machine to machine.
|
||||
`CHATGPT_SESSION_TOKEN_1` is optional; both layouts authenticate, because the
|
||||
server reassembles a complete value sent under the `.0` name.
|
||||
|
||||
What does *not* work is sending a **partial** chunk — `.0` on its own when a `.1`
|
||||
exists. That fails silently: `/api/auth/session` answers HTTP 200 with no
|
||||
`accessToken` rather than an error. If you see two cookies, copy both.
|
||||
|
||||
**Claude:** go to `https://claude.ai` → find `sessionKey` → copy Value
|
||||
|
||||
### When Tokens Expire
|
||||
|
||||
An expired token shows up as an authentication error naming the cookie to
|
||||
refresh and how. The status differs by provider — ChatGPT reports 401, while
|
||||
claude.ai reports **403 "Invalid authorization"** (`account_session_invalid`) —
|
||||
so don't read a 403 from Claude as a permissions problem with your account.
|
||||
|
||||
To refresh:
|
||||
- Re-run the `auth` wizard: `ai-chat-exporter auth`
|
||||
- Or manually update the value in your `.env` file
|
||||
|
||||
`ai-chat-exporter doctor` is the quickest check: the "token set" rows only test
|
||||
that a value is present, so an expired credential passes those and fails on the
|
||||
"API reachable" row.
|
||||
|
||||
---
|
||||
|
||||
## The `auth` Command
|
||||
|
||||
The easiest way to configure tokens is the interactive wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
This walks you through finding your token, validates it, shows the expiry date (ChatGPT only), and offers to write it to your `.env` automatically. Tokens are never echoed to the terminal.
|
||||
|
||||
---
|
||||
|
||||
## ChatGPT Projects
|
||||
|
||||
ChatGPT project conversations are stored separately from your main conversation list and require extra configuration.
|
||||
|
||||
### Finding your project IDs
|
||||
|
||||
The quickest way is to let the exporter find them:
|
||||
|
||||
```
|
||||
ai-chat-exporter projects
|
||||
```
|
||||
|
||||
It lists every project your conversations actually belong to, marks which are
|
||||
missing from `.env`, and prints a paste-ready `CHATGPT_PROJECT_IDS=` line
|
||||
(`--write` updates `.env` for you). If your ChatGPT account does not include
|
||||
the project on conversation summaries, add `--deep` and it reads each
|
||||
conversation's detail instead — slower, one request per conversation, but
|
||||
complete.
|
||||
|
||||
Why it matters: project attribution is resolved from each conversation's own
|
||||
`gizmo_id`, so exports file correctly whether or not a project is configured.
|
||||
But the *listing* pass still needs `CHATGPT_PROJECT_IDS` — conversations that
|
||||
live only inside a project never appear in the default conversation list, so an
|
||||
unlisted project's chats are never fetched at all. Every export run also names
|
||||
any unconfigured project it encounters.
|
||||
|
||||
To find them by hand instead:
|
||||
|
||||
1. Open ChatGPT and click a Project in the left sidebar
|
||||
2. Look at the browser URL — it will look like:
|
||||
`https://chatgpt.com/g/g-p-68c2b2b3037c8191890036fb4ae3ed9f-my-project/project`
|
||||
3. Copy the `g-p-…` part (everything up to but not including the slug after the second `-`)
|
||||
|
||||
Add all your project IDs to `.env` as a comma-separated list:
|
||||
|
||||
```
|
||||
CHATGPT_PROJECT_IDS=g-p-68c2b2b3037c8191890036fb4ae3ed9f,g-p-anotherprojectid
|
||||
```
|
||||
|
||||
The `auth` wizard can also guide you through this step interactively.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Sessions
|
||||
|
||||
The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider claude-code
|
||||
ai-chat-exporter joplin --provider claude-code
|
||||
```
|
||||
|
||||
Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Sessions become notes under their own top-level **`AI-ClaudeCode`** Joplin notebook, in a sub-notebook per launch folder. The provider is included in `--provider all` whenever a sessions directory exists.
|
||||
|
||||
**Subagents.** Claude Code stores subagent (Task-tool) transcripts as separate files under `<session>/subagents/`; each is folded into its parent session inline, as a collapsible `<details>` block labeled with the subagent's type and description (its own tool traffic is collapsed like the main dialogue).
|
||||
|
||||
**Repo tags.** Sessions launched from a workspace root all share one folder-named notebook, so titles carry the repos each session touched — `Resume StartWRT project work [start-technologies]` — for at-a-glance scanning and search. A file's repo is the git repository it lives in (nearest ancestor with a `.git`), resolved from the tool paths in the transcript, so work is tagged wherever it happened — even across workspaces — and config/one-off files are ignored (they aren't repos). Repos are frequency-ordered and capped at 3. To never tag specific repos, set `CLAUDE_CODE_REPO_TAG_IGNORE` (comma-separated). Note this reads your current git layout, so a repo you later delete or move drops from the tag on re-export.
|
||||
|
||||
**Multiple locations.** By default the provider scans `~/.claude/projects/` plus `$CLAUDE_CONFIG_DIR/projects` when `CLAUDE_CONFIG_DIR` is set. To scan additional roots (e.g. other machines' sessions copied onto this box), set `CLAUDE_CODE_DIR` to a `:`-separated list:
|
||||
|
||||
```bash
|
||||
CLAUDE_CODE_DIR="$HOME/.claude/projects:/mnt/backup/laptop/.claude/projects"
|
||||
```
|
||||
|
||||
Sessions from all roots are merged by folder (no per-machine label); if the same session UUID appears in two roots, the newer copy wins. Note the exporter only sees this machine's disk and cannot recover sessions Claude Code has already pruned — run it regularly.
|
||||
|
||||
---
|
||||
|
||||
## Codex Sessions
|
||||
|
||||
The `codex` provider archives your local [Codex CLI](https://chatgpt.com/codex) agent transcripts — same deal as Claude Code: no tokens, no API, no ToS exposure. Rollout files are read from `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider codex
|
||||
ai-chat-exporter joplin --provider codex
|
||||
```
|
||||
|
||||
Exports are prose-only by default, with the same placeholder format as Claude Code, into their own top-level **`AI-Codex`** Joplin notebook. Repo tags work the same way (`CODEX_REPO_TAG_IGNORE` to suppress), and additional roots can be scanned with `CODEX_DIR` (`:`-separated); `CODEX_HOME`'s `sessions/` is picked up automatically when that variable is set.
|
||||
|
||||
Three things differ from Claude Code, all forced by how Codex stores its data:
|
||||
|
||||
**Reasoning cannot be exported.** Codex encrypts it at rest — every reasoning record carries `encrypted_content` with no plaintext summary in any layer of the file. It is always dropped and counted; `EXPORTER_HIDDEN_CONTENT=full` cannot bring it back.
|
||||
|
||||
**Incomplete tool calls are reported.** Codex writes each session twice in one file: its own typed items (what actually ran) and the raw model-facing wire format (everything attempted). The provider reads the typed layer — it is already decoded, and it omits harness plumbing that would otherwise need stripping — but cross-checks the raw layer for calls that never produced a result, so placeholders read `3 calls: exec_command ×3 (+2 did not complete)`. Those are commands that failed to launch, that you aborted, or that were still running when the turn ended.
|
||||
|
||||
**Cloud tasks are out of scope.** `codex cloud` tasks run server-side and are reachable at `chatgpt.com/backend-api/api/codex/tasks`, but local CLI sessions are never uploaded there, so the cloud API is not an alternative source for these transcripts and this provider stays entirely offline. If you start using `codex cloud exec`, those transcripts would be cloud-only and would need separate work.
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
# Running on a Schedule
|
||||
|
||||
Installing the daily run on Linux and Windows, what to know before relying on
|
||||
it, and how results reach your phone.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [CLI Reference](cli.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Scheduling a Daily Run
|
||||
|
||||
`sync` chains `export` then `joplin` in one invocation, which is what a scheduler
|
||||
wants — one command, and a meaningful exit code so a failed run is visible
|
||||
instead of silent.
|
||||
|
||||
```bash
|
||||
./ai-chat-exporter sync # every configured provider
|
||||
./ai-chat-exporter sync --provider codex # just one
|
||||
```
|
||||
|
||||
### Linux (systemd user timer)
|
||||
|
||||
```bash
|
||||
./scheduling/install-systemd-timer.sh --provider claude-code --provider codex
|
||||
./scheduling/install-systemd-timer.sh --uninstall
|
||||
```
|
||||
|
||||
Defaults to 09:00 daily (`--time 21:30` to change). `Persistent=true` means a
|
||||
machine that was off at the scheduled time runs the archive at next boot rather
|
||||
than skipping the day. To archive while logged out, `loginctl enable-linger $USER`.
|
||||
|
||||
Check on it with `systemctl --user list-timers aichat-sync.timer` and
|
||||
`journalctl --user -u aichat-sync.service -n 50`.
|
||||
|
||||
### Windows (Task Scheduler)
|
||||
|
||||
```powershell
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Provider chatgpt,claude
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Unregister
|
||||
```
|
||||
|
||||
Per-user task, no admin rights needed. `-StartWhenAvailable` is the counterpart
|
||||
of systemd's `Persistent=true`. The task runs `scheduling\run-sync.ps1` with its
|
||||
window hidden; Task Scheduler keeps no output, so each run's output is appended
|
||||
to `cache\logs\scheduled-sync.log` (rolled over to `.1` past 1 MB).
|
||||
|
||||
Check on it with `Get-ScheduledTaskInfo -TaskName AiChatExporterSync` (look at
|
||||
`LastTaskResult`: 0 is success) and
|
||||
`Get-Content cache\logs\scheduled-sync.log -Tail 50`.
|
||||
|
||||
### What to know before you rely on it
|
||||
|
||||
**List only the providers that work unattended on that machine.** The local
|
||||
providers (`claude-code`, `codex`) need no credentials and always work. The web
|
||||
providers depend on a session token that expires and can only be refreshed by
|
||||
hand via DevTools — so on a machine where that token is stale, scheduling them
|
||||
means a failed run every single day, which is a good way to learn to ignore
|
||||
failures you actually want to see. That is why `--provider` is repeatable in both
|
||||
installers: schedule the coding machine for `claude-code` + `codex`, the browser
|
||||
machine for `chatgpt` + `claude`.
|
||||
|
||||
**Both installers pass `--joplin-optional`.** If Joplin desktop isn't running,
|
||||
the sync warns instead of failing: the export has already captured the local
|
||||
transcripts (the part that can disappear), and the notes are rebuilt from the
|
||||
cache on the next run that finds Joplin up.
|
||||
|
||||
**One provider failing does not skip the rest.** Both installers loop over the
|
||||
providers in a single action rather than one action each, because systemd
|
||||
`oneshot` stops at the first failing `ExecStart` and Task Scheduler reports only
|
||||
the last action's result. Every provider is attempted; the run still exits
|
||||
non-zero if any failed. The loop is `scheduling/run-sync.sh` on Linux (the
|
||||
unit's `ExecStart`) and `scheduling\run-sync.ps1` on Windows (the task's action).
|
||||
|
||||
### Getting notified
|
||||
|
||||
A scheduled run is silent by default. Output goes to three pull-only places: the
|
||||
exporter's own log (`cache/logs/exporter.log`), the systemd journal on Linux
|
||||
(`journalctl --user -u aichat-sync.service`), and on Windows Task Scheduler's
|
||||
`LastTaskResult` plus `cache\logs\scheduled-sync.log`.
|
||||
|
||||
To have runs report back, set an [ntfy](https://ntfy.sh) topic in `.env`:
|
||||
|
||||
```bash
|
||||
NTFY_TOPIC=my-archive-topic
|
||||
```
|
||||
|
||||
Then check it works before waiting on a scheduled run:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter notify # show settings
|
||||
ai-chat-exporter notify --test # send a test push
|
||||
```
|
||||
|
||||
`sync` pushes a result whenever a topic is configured — a success carries the
|
||||
per-provider counts at low/default priority, a failure carries the reason at
|
||||
high priority with an alert tag, so a failed archive is distinguishable from a
|
||||
quiet one on your phone. `NTFY_NOTIFY=failure` notifies only on failure; `off`
|
||||
disables it; `--notify` / `--no-notify` override per run.
|
||||
|
||||
`sync` can only push from the end of a run it finished. A crash, an exit before
|
||||
the sync starts (the ToS gate, a cache error) or a launcher that can't build its
|
||||
venv sends nothing — and since each provider pushes separately, the ones that
|
||||
succeeded still say "OK", so a dead provider looks like a quiet day. The
|
||||
scheduler wrappers (`scheduling/run-sync.sh`, `scheduling\run-sync.ps1`) close
|
||||
that gap: any run that exits non-zero without the
|
||||
app having reported it gets a high-priority **FAILED** push naming the provider
|
||||
and, for a crash, the exception's class (`claude-code: crashed (RecursionError)`)
|
||||
— the class only, never its message, which can carry a conversation title. The
|
||||
traceback is in the journal on Linux and in `cache\logs\scheduled-sync.log` on
|
||||
Windows.
|
||||
|
||||
The message includes the **machine name**, which matters because both machines
|
||||
archive into one topic. It contains counts only — never conversation titles. A
|
||||
topic on public ntfy.sh is readable by anyone who knows its name, so if you want
|
||||
it private, self-host (`NTFY_SERVER`) or use an access-controlled topic with
|
||||
`NTFY_TOKEN`.
|
||||
|
||||
A notification is never fatal: if ntfy is unreachable, the run logs a warning and
|
||||
still reports its real exit code.
|
||||
|
||||
**Acknowledge the ToS notice once, interactively.** It's stored in the cache
|
||||
manifest per machine. Until then a scheduled run exits 1 with an explanation
|
||||
rather than hanging on a prompt no one can answer.
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
# Troubleshooting
|
||||
|
||||
What the cache does, what each failure means, and how to file a bug report.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [CLI Reference](cli.md) · [Scheduling](scheduling.md)
|
||||
|
||||
---
|
||||
|
||||
## How the Cache Works
|
||||
|
||||
The cache manifest lives at `cache/manifest.json` (inside the install directory) and records every exported conversation: its title, project, `updated_at` timestamp, output file path, and (after Joplin sync) the Joplin note ID.
|
||||
|
||||
On every `export` run:
|
||||
1. Fetch the full conversation list from the provider
|
||||
2. Compare each conversation's `updated_at` against the manifest
|
||||
3. Export only conversations that are new or have been updated
|
||||
4. Write each successfully exported conversation to the manifest **immediately** (not batched)
|
||||
|
||||
On every `joplin` run:
|
||||
1. Read the manifest to find conversations not yet synced to Joplin, or re-exported since last sync
|
||||
2. Push each pending Markdown file to Joplin (create or update)
|
||||
3. Store the Joplin note ID in the manifest so subsequent runs update rather than duplicate
|
||||
|
||||
**This design makes every run inherently resumable.** If the tool is interrupted for any reason — rate limit, network drop, Ctrl+C, crash — simply re-run the same command. It will skip already-processed conversations and continue from where it stopped.
|
||||
|
||||
A conversation whose assets failed transiently also carries `pending_media` (the
|
||||
asset refs still missing) and `pending_media_attempts`. While `pending_media` is
|
||||
set and the attempt cap is not reached, step 2 treats the conversation as stale
|
||||
so the next run re-fetches those assets — see
|
||||
[Embedded images and media](#embedded-images-and-media). A run that recovers them
|
||||
clears the key.
|
||||
|
||||
To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
|
||||
|
||||
To re-export a single conversation, delete its entry from `cache/manifest.json`
|
||||
and re-run. (There is deliberately no `cache --reset --conversation` flag; see
|
||||
`FUTURE.md`.)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `Authentication failed` (401, or 403 from Claude)
|
||||
Your session token has expired.
|
||||
- Run `ai-chat-exporter auth` to get a new token interactively
|
||||
- Or manually copy a fresh cookie value into your `.env` file
|
||||
|
||||
Note: neither token's expiry can be read client-side. Claude's `sessionKey` is an opaque string, and claude.ai reports an invalid one as **403** "Invalid authorization" (`account_session_invalid`), not 401. ChatGPT's token is an encrypted JWE; `doctor` reads the `error` field of `/api/auth/session` instead. See [When Tokens Expire](providers.md#when-tokens-expire).
|
||||
|
||||
### `429 Rate Limited`
|
||||
The tool automatically pauses, saves progress, and exits with a clear message showing how many conversations were exported vs remaining. Just re-run the same export command to resume — the cache picks up exactly where it left off.
|
||||
|
||||
### `HTTP 500` / `503` on an asset download
|
||||
The provider's asset storage, not your setup. Retries and the cross-run recovery
|
||||
described in [Embedded images and media](#embedded-images-and-media) handle this;
|
||||
the run reports `media failed` and the next run re-fetches. Measured 2026-10-20:
|
||||
two assets lost this way to a 500 and a 503 both downloaded cleanly an hour later.
|
||||
|
||||
### Joplin: "JOPLIN_API_TOKEN is not set"
|
||||
You need to configure the token before running the `joplin` command:
|
||||
1. Open Joplin desktop
|
||||
2. Go to Tools → Options → Web Clipper
|
||||
3. Enable the Web Clipper service
|
||||
4. Copy the Authorization token shown on that page
|
||||
5. Add `JOPLIN_API_TOKEN=<token>` to your `.env` file
|
||||
|
||||
### Joplin: "Joplin is not responding"
|
||||
Joplin desktop must be running when you run the `joplin` command. The Web Clipper service shuts down when Joplin is closed.
|
||||
|
||||
### Joplin: "Joplin rejected the API token (HTTP 401)"
|
||||
The token in `JOPLIN_API_TOKEN` doesn't match what Joplin expects. Get a fresh token from Joplin → Tools → Options → Web Clipper → Authorization token.
|
||||
|
||||
### Joplin: note timed out
|
||||
If you see a timeout error, Joplin took longer than `JOPLIN_REQUEST_TIMEOUT` seconds (default: 30) to respond. Possible causes:
|
||||
- The conversation is very large and Joplin is slow to index it
|
||||
- Joplin is busy syncing or loading a large library
|
||||
- Joplin has frozen — try restarting it
|
||||
|
||||
To increase the timeout: add `JOPLIN_REQUEST_TIMEOUT=60` to your `.env`.
|
||||
|
||||
### ChatGPT project conversations not appearing
|
||||
Make sure you've added the project IDs to `CHATGPT_PROJECT_IDS` in your `.env`. See [ChatGPT Projects](providers.md#chatgpt-projects) for how to find them. Project conversations are not included in the default conversation listing — they must be fetched separately.
|
||||
|
||||
### Schema warnings in logs (`Unexpected API response shape`)
|
||||
The provider's internal API may have changed. Run with `--debug`, sanitize the output (remove any personal content), and check the project's GitHub Issues for known fixes.
|
||||
|
||||
### Non-text content warnings
|
||||
Since v0.4.0, rich content is preserved as typed blocks in the export. ChatGPT voice transcripts render as text and audio assets as `📎 File attached` placeholders with size and duration metadata. Anything the extractor doesn't recognise renders as a visible `> ⚠️ Unsupported content` block naming the type and observed keys, *and* increments a counter in the post-export summary so you can tell whether real content is being silently skipped.
|
||||
|
||||
### Embedded images and media
|
||||
Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as ``. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
|
||||
|
||||
A failed asset **never fails the export** — the conversation is archived with its
|
||||
text placeholder, and the failure is tallied as `media failed` in the run summary
|
||||
and in the push notification. What happens next depends on *why* it failed:
|
||||
|
||||
| Bucket | Meaning | Retried? |
|
||||
|---|---|---|
|
||||
| `expired-or-missing` | 404 — the upload was deleted server-side | No. The bytes are gone. |
|
||||
| `forbidden` | 403 — metadata survives, bytes do not (see the 2026-08-17 investigation in `CHANGELOG.md`) | No. |
|
||||
| `download-error` | 5xx, a timeout, a network blip | **Yes** — within the run, then on up to 3 later runs |
|
||||
| `skipped-provider-unwell` | the run's circuit breaker was open | **Yes**, on a later run |
|
||||
| `unparseable-ref` / `write-error` | local problem, not the provider's | No |
|
||||
|
||||
**Within a run**, an asset download is two requests — one for a signed URL, one
|
||||
for the bytes — and each is retried on a 5xx: immediately once, then after about
|
||||
2, 4, 8 and 16 seconds with jitter, honouring `Retry-After` when the server sends
|
||||
one. Retries for a single asset are capped by a wall-clock budget so a hanging
|
||||
endpoint cannot consume the run's whole execution window.
|
||||
|
||||
**Across runs**, an asset that failed transiently is recorded against its
|
||||
conversation in the cache manifest, and the next run re-exports that conversation
|
||||
to re-fetch it. Only the missing asset is fetched — media already on disk is
|
||||
reused, and Joplin resource IDs are keyed by path, so the re-sync uploads just
|
||||
the new file. After 3 unsuccessful runs the asset is treated as lost and the
|
||||
conversation stops being re-exported.
|
||||
|
||||
**Circuit breaker.** If 5 transient failures land back to back, asset downloads
|
||||
stop for the rest of the run and the remaining assets are recorded for a later
|
||||
attempt. A provider's storage having a bad day should cost one quiet morning, not
|
||||
a run that overruns its two-hour scheduled window. Permanent failures never count
|
||||
toward this — a deleted upload says nothing about the provider's health.
|
||||
|
||||
### Tool output / hidden context collapsed
|
||||
Since v0.6.0, content that was invisible in the ChatGPT web UI is collapsed by default to one-line placeholders like `> 🔧 Tool output — file_search (15.0 KB) — omitted`. This is ChatGPT's file-retrieval tool re-injecting your attached files on every run — it can be 90% of a conversation's bytes while containing none of the dialogue. Custom Instructions collapse the same way (`> ℹ️ Hidden context`). The post-export summary lists every collapsed message by origin with the total KB omitted. To keep everything, set `EXPORTER_HIDDEN_CONTENT=full` in `.env` or pass `--hidden-content full`.
|
||||
|
||||
### Empty export / all conversations skipped
|
||||
No new or updated conversations since your last run. To verify: `ai-chat-exporter cache --show`. To force a full re-export: `ai-chat-exporter cache --clear`.
|
||||
|
||||
### Filing a bug report
|
||||
1. Run with `--debug`: `ai-chat-exporter export --debug 2>&1 | tee debug.log`
|
||||
2. Remove any personal conversation content from `debug.log`
|
||||
3. Open a GitHub Issue with the sanitized log and the exact command you ran
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "ai-chat-exporter"
|
||||
version = "0.9.0"
|
||||
version = "1.0.0"
|
||||
description = "Archive ChatGPT, Claude, Claude Code and Codex conversation history to Markdown for personal backup in Joplin"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
|
||||
Reference in New Issue
Block a user