release: v1.0.0, and split the README into docs/
The archive does what it was built to do, unattended, on both machines: four providers, incremental and resumable, Markdown plus Joplin, on a schedule, with results pushed to a phone and failures that announce themselves. The one item left on the roadmap — a StartOS service that consolidates both machines into one corpus — is an optional convenience, not a gap. Hence 1.0.0 rather than another 0.x. The code in this release shipped across b6ce636..8c3e00a: the claude.ai 403 session-invalid fix, the fork-subagent recursion fix, the FAILED-push backstops on both platforms, and asset-download retries with cross-run recovery, a circuit breaker and media counts in the notification. This commit cuts the release and pays a documentation debt. The README had reached 939 lines / 6,490 words / 48 KB / 96 headings, doing four unrelated jobs at once, and FUTURE.md asked for the split to ride along with the next substantial README edit rather than break every existing anchor for its own sake. A release is that edit. README.md is now 303 lines and keeps only what you need to install and configure the tool; the rest moved verbatim into docs/providers.md, docs/cli.md, docs/scheduling.md and docs/troubleshooting.md. No section was rewritten or dropped and every original heading still exists, so each anchor still resolves somewhere — but bookmarks into moved sections now need the docs/ path, which is the cost this deferral was protecting against. .gitignore needed the fix FUTURE.md predicted: the blanket *.md rule (exported conversations are Markdown and may be private) matched all four new pages, and git check-ignore confirmed every one would have been silently ignored. !docs/*.md is added with a comment saying why. Gitea's rendering of relative docs/ links cannot be checked before pushing, so it still needs confirming after. Also corrects two stale claims in the changelog: the Windows backstop is no longer "not yet run under Windows PowerShell 5.1 or Task Scheduler" (confirmed on winmini today, LastTaskResult 0 — only the FAILED push itself remains unexercised there), and the test count is 414, not 409. 414 tests passing. No code changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012gCWSknsWvhLnMFwa7qnGp
This commit is contained in:
JesseMarkowitz
co-authored by
Claude Opus 5
Claude Opus 5
parent
8c3e00a520
commit
04ebb8987c
+254
@@ -0,0 +1,254 @@
|
||||
# CLI Reference
|
||||
|
||||
Every command, its flags and what it does.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [Scheduling](scheduling.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Global flags
|
||||
|
||||
```
|
||||
--verbose / -v DEBUG output to console
|
||||
--quiet / -q WARNING and above only
|
||||
--debug DEBUG + full tracebacks + redacted API response bodies
|
||||
--no-log-file Disable file logging
|
||||
--version Print version and exit
|
||||
```
|
||||
|
||||
## `auth` — Interactive token setup
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth # manual wizard (DevTools flow)
|
||||
```
|
||||
|
||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut, then writes the values to `.env`.
|
||||
|
||||
## `doctor` — Health check
|
||||
|
||||
```bash
|
||||
ai-chat-exporter doctor
|
||||
```
|
||||
|
||||
Checks: token presence, JWT validity and expiry, directory permissions, disk space, live API reachability. Exits with code 0 if all pass, 1 if any fail.
|
||||
|
||||
## `export` — Export conversations
|
||||
|
||||
```bash
|
||||
# Export everything (new/updated only)
|
||||
ai-chat-exporter export
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter export --provider claude
|
||||
|
||||
# JSON output
|
||||
ai-chat-exporter export --format json
|
||||
|
||||
# Both Markdown and JSON
|
||||
ai-chat-exporter export --format both
|
||||
|
||||
# Only conversations updated since a date
|
||||
ai-chat-exporter export --since 2024-06-01
|
||||
|
||||
# Only conversations in a specific project (case-insensitive substring)
|
||||
ai-chat-exporter export --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter export --project none
|
||||
|
||||
# Write to a custom directory
|
||||
ai-chat-exporter export --output /path/to/my/notes
|
||||
|
||||
# Preview without writing anything
|
||||
ai-chat-exporter export --dry-run
|
||||
```
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--force`, `--dry-run`
|
||||
|
||||
**Re-rendering the whole archive after an upgrade.** New formatting or features (collapse policy, media downloads) only change conversations as they're re-exported. To re-render everything you already have, use `--force` — it re-exports every conversation even if unchanged, **without** `cache --clear`, so your Joplin note links are preserved (a later `joplin` run updates the existing notes instead of duplicating them).
|
||||
|
||||
A force re-render runs as a tracked "campaign": each run re-renders the least-recently-exported conversations, the "still to go" count shrinks each run, and finished providers do no further work. The simplest approach is one uncapped pass:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force # re-renders everything in one paced pass
|
||||
ai-chat-exporter joplin # update notes + upload media
|
||||
```
|
||||
|
||||
Or spread the load with `--max-conversations`, re-running until it reports "Force re-render complete":
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --force --max-conversations 50 # repeat until complete
|
||||
```
|
||||
|
||||
## `list` — List conversations
|
||||
|
||||
```bash
|
||||
# List all conversations for all providers
|
||||
ai-chat-exporter list
|
||||
|
||||
# Single provider
|
||||
ai-chat-exporter list --provider chatgpt
|
||||
|
||||
# Filter by project
|
||||
ai-chat-exporter list --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
ai-chat-exporter list --project none
|
||||
```
|
||||
|
||||
Fetches and displays all conversations without exporting them. Useful for verifying what the tool can see before running an export.
|
||||
|
||||
## `joplin` — Sync to Joplin
|
||||
|
||||
```bash
|
||||
# Sync all pending conversations to Joplin
|
||||
ai-chat-exporter joplin
|
||||
|
||||
# Preview what would be synced without sending anything
|
||||
ai-chat-exporter joplin --dry-run
|
||||
|
||||
# Sync a single provider
|
||||
ai-chat-exporter joplin --provider chatgpt
|
||||
|
||||
# Sync only conversations in a specific project
|
||||
ai-chat-exporter joplin --project "learning python"
|
||||
|
||||
# Sync only conversations outside any project
|
||||
ai-chat-exporter joplin --project none
|
||||
```
|
||||
|
||||
Reads the local export cache and pushes each exported Markdown file to Joplin as a note. Notebooks are created automatically. Re-running is safe — notes are updated (not duplicated).
|
||||
|
||||
**Prerequisites:**
|
||||
1. Run `export` first to generate the Markdown files
|
||||
2. Open Joplin → Tools → Options → Web Clipper → enable the service
|
||||
3. Copy the Authorization token and add `JOPLIN_API_TOKEN=<token>` to your `.env`
|
||||
4. Joplin desktop must be open when you run this command
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--project NAME`, `--dry-run`
|
||||
|
||||
## `sync` — Export and sync in one run
|
||||
|
||||
```bash
|
||||
# The whole archive run: export, then push to Joplin
|
||||
ai-chat-exporter sync
|
||||
|
||||
# One provider
|
||||
ai-chat-exporter sync --provider codex
|
||||
|
||||
# Export only; don't touch Joplin
|
||||
ai-chat-exporter sync --skip-joplin
|
||||
|
||||
# Joplin being closed is a warning, not a failure (used by the schedulers)
|
||||
ai-chat-exporter sync --joplin-optional
|
||||
```
|
||||
|
||||
Equivalent to `export` followed by `joplin` with the same `--provider`. Intended
|
||||
for scheduled runs — see [Scheduling a Daily Run](scheduling.md#scheduling-a-daily-run).
|
||||
|
||||
Unlike the individual commands, `sync` sets a **meaningful exit code**: non-zero
|
||||
if any conversation failed to export or any note failed to sync. A provider whose
|
||||
listing call fails outright (an expired web session token being the usual cause)
|
||||
counts its whole batch as failed. A provider that is simply unconfigured, or that
|
||||
had nothing new, is ordinary success. `export` on its own always exits 0, which
|
||||
is fine when you're reading the summary table and useless to a scheduler.
|
||||
|
||||
`--joplin-optional` downgrades an unreachable Joplin to a warning: the export has
|
||||
already captured the local transcripts, and the notes are rebuilt from the cache
|
||||
by the next run that finds Joplin open.
|
||||
|
||||
Options: `--provider [chatgpt|claude|claude-code|codex|all]`, `--since YYYY-MM-DD`, `--hidden-content [full|placeholder|omit]`, `--max-conversations N`, `--skip-joplin`, `--joplin-optional`, `--notify/--no-notify`, `--dry-run`
|
||||
|
||||
Note this is a deliberate subset of `export`'s options — `--format`, `--output`,
|
||||
`--project`, `--download-media` and `--force` are not passed through. Use
|
||||
`export` directly for those. (`--download-media` still applies from `.env`; the
|
||||
flag is only a per-run override.)
|
||||
|
||||
## `notify` — Push-notification settings and test
|
||||
|
||||
```bash
|
||||
# Show the current settings
|
||||
ai-chat-exporter notify
|
||||
|
||||
# Send a test push to confirm the topic works
|
||||
ai-chat-exporter notify --test
|
||||
```
|
||||
|
||||
Shows the resolved ntfy configuration and which machine name will appear in the
|
||||
title. See [Getting notified](scheduling.md#getting-notified) for what a scheduled run sends.
|
||||
|
||||
Options: `--test`
|
||||
|
||||
## `prune` — Delete stale export files
|
||||
|
||||
```bash
|
||||
# Preview what would be deleted
|
||||
ai-chat-exporter prune --dry-run
|
||||
|
||||
# Delete (asks for confirmation; -y skips the prompt)
|
||||
ai-chat-exporter prune
|
||||
```
|
||||
|
||||
Deletes export files no longer referenced by the cache manifest — leftovers
|
||||
from old folder layouts or fixed bugs that would otherwise double-sync into
|
||||
Joplin. Refuses to run when the manifest is empty (e.g. right after
|
||||
`cache --clear`) so it can never wipe a freshly cleared archive. The `doctor`
|
||||
command separately verifies that every manifest entry's file exists on disk.
|
||||
|
||||
## `projects` — Discover ChatGPT project IDs
|
||||
|
||||
```bash
|
||||
# List the projects your conversations belong to
|
||||
ai-chat-exporter projects
|
||||
|
||||
# Also inspect conversations whose listing entry doesn't name a project
|
||||
ai-chat-exporter projects --deep
|
||||
|
||||
# Write the discovered IDs straight into .env
|
||||
ai-chat-exporter projects --write
|
||||
```
|
||||
|
||||
`CHATGPT_PROJECT_IDS` is maintained by hand, and a project missing from it is
|
||||
invisible to the listing pass — conversations that live *only* inside that
|
||||
project are never fetched at all. This reports every project your conversations
|
||||
belong to, marks the ones absent from `.env`, and prints a paste-ready line.
|
||||
|
||||
`--deep` fetches each conversation's detail when the listing doesn't name its
|
||||
project: complete, but one request per conversation, so it's slow.
|
||||
|
||||
Options: `--deep`, `--write`
|
||||
|
||||
## `canary` — Check for provider API drift
|
||||
|
||||
```bash
|
||||
ai-chat-exporter canary
|
||||
ai-chat-exporter canary --provider chatgpt
|
||||
```
|
||||
|
||||
The web providers are undocumented internal APIs that can change shape without
|
||||
notice, and the failure mode is silent — a renamed field means content is
|
||||
quietly dropped rather than an error being raised. The canary fetches one
|
||||
listing page and one conversation per provider and asserts only the fields the
|
||||
normalizer actually depends on.
|
||||
|
||||
Findings are `ERROR` (a load-bearing field is missing or mistyped — the parser
|
||||
will break or silently lose data) or `WARN` (something unfamiliar appeared;
|
||||
worth investigating, not necessarily broken). **Exits non-zero on any ERROR**,
|
||||
so it can be scheduled or run in CI. Local providers have no remote schema and
|
||||
are not probed.
|
||||
|
||||
Options: `--provider [chatgpt|claude|all]`
|
||||
|
||||
## `cache` — Manage the sync manifest
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
ai-chat-exporter cache --show
|
||||
|
||||
# Clear all cached entries (forces full re-export next run)
|
||||
ai-chat-exporter cache --clear
|
||||
|
||||
# Clear a single provider
|
||||
ai-chat-exporter cache --clear --provider claude
|
||||
```
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
# Providers and Tokens
|
||||
|
||||
How to obtain and refresh the session tokens for the two web providers, and the
|
||||
per-provider setup for ChatGPT Projects, Claude Code and Codex.
|
||||
|
||||
← [README](../README.md) · [Configuration](../README.md#env-setup) · [CLI Reference](cli.md) · [Scheduling](scheduling.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Getting Your Session Tokens
|
||||
|
||||
Session tokens are how your browser stays logged in. This tool uses them to access your chat history on your behalf.
|
||||
|
||||
Tokens are entered manually — copy them from your browser's DevTools and run the wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
The wizard detects your OS, shows the correct DevTools shortcut, and writes the values to `.env` without echoing them to the terminal.
|
||||
|
||||
> **Why no auto-extraction from the browser?** Modern Chromium browsers (Chrome 127+, current Brave) encrypt cookies on Windows with **App-Bound Encryption**: the key is bound to the browser through a SYSTEM-level service, so reading cookies off disk requires Administrator rights and a PsExec-style SYSTEM impersonation that antivirus flags as credential theft — and it still fails on Brave specifically. There is no reliable, non-invasive way to pull these cookies automatically, so the manual DevTools flow below is the supported path.
|
||||
|
||||
### Token Lifetimes
|
||||
|
||||
| Provider | Cookie Name | Lifetime | Expiry Detection |
|
||||
|----------|-------------|----------|-----------------|
|
||||
| ChatGPT | `__Secure-next-auth.session-token` (split into `.0` + `.1` when over ~4KB) | refresh ~weekly | `error` field of `/api/auth/session` — `doctor` reports "ChatGPT token active". The token is an encrypted JWE, so its `exp` is **not** readable client-side, and the `expires` field is a misleading rolling window; the `error` (`RefreshAccessTokenError` when dead) is the honest signal. |
|
||||
| Claude | `sessionKey` | ~30 days | Opaque token — only detectable from an API rejection. claude.ai answers an invalid session with **403** `permission_error` / `account_session_invalid`, **not** 401; `doctor` reports it on the "Claude API reachable" row. |
|
||||
|
||||
### Finding Tokens in Chrome DevTools
|
||||
|
||||
1. Open the provider's website and make sure you're logged in
|
||||
2. Press **F12** (Windows/Linux) or **Cmd+Option+I** (macOS) to open DevTools
|
||||
3. Click the **Application** tab
|
||||
4. In the left panel, expand **Cookies** and click the site URL
|
||||
5. Find the cookie by name and copy its **Value**
|
||||
|
||||
**ChatGPT:** go to `https://chatgpt.com` → find the session token cookie. You will
|
||||
see **one of two layouts**, depending on how large your session token is:
|
||||
|
||||
- **One cookie**, `__Secure-next-auth.session-token` — copy Value → `CHATGPT_SESSION_TOKEN`, and leave `CHATGPT_SESSION_TOKEN_1` empty.
|
||||
- **Two cookies**, `__Secure-next-auth.session-token.0` and `.1` — copy `.0` (starts with `eyJ`) → `CHATGPT_SESSION_TOKEN`, and `.1` → `CHATGPT_SESSION_TOKEN_1`.
|
||||
|
||||
Chrome splits a cookie only when it exceeds ~4KB, so a larger session is chunked
|
||||
and a smaller one is not — the same account can differ from machine to machine.
|
||||
`CHATGPT_SESSION_TOKEN_1` is optional; both layouts authenticate, because the
|
||||
server reassembles a complete value sent under the `.0` name.
|
||||
|
||||
What does *not* work is sending a **partial** chunk — `.0` on its own when a `.1`
|
||||
exists. That fails silently: `/api/auth/session` answers HTTP 200 with no
|
||||
`accessToken` rather than an error. If you see two cookies, copy both.
|
||||
|
||||
**Claude:** go to `https://claude.ai` → find `sessionKey` → copy Value
|
||||
|
||||
### When Tokens Expire
|
||||
|
||||
An expired token shows up as an authentication error naming the cookie to
|
||||
refresh and how. The status differs by provider — ChatGPT reports 401, while
|
||||
claude.ai reports **403 "Invalid authorization"** (`account_session_invalid`) —
|
||||
so don't read a 403 from Claude as a permissions problem with your account.
|
||||
|
||||
To refresh:
|
||||
- Re-run the `auth` wizard: `ai-chat-exporter auth`
|
||||
- Or manually update the value in your `.env` file
|
||||
|
||||
`ai-chat-exporter doctor` is the quickest check: the "token set" rows only test
|
||||
that a value is present, so an expired credential passes those and fails on the
|
||||
"API reachable" row.
|
||||
|
||||
---
|
||||
|
||||
## The `auth` Command
|
||||
|
||||
The easiest way to configure tokens is the interactive wizard:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
This walks you through finding your token, validates it, shows the expiry date (ChatGPT only), and offers to write it to your `.env` automatically. Tokens are never echoed to the terminal.
|
||||
|
||||
---
|
||||
|
||||
## ChatGPT Projects
|
||||
|
||||
ChatGPT project conversations are stored separately from your main conversation list and require extra configuration.
|
||||
|
||||
### Finding your project IDs
|
||||
|
||||
The quickest way is to let the exporter find them:
|
||||
|
||||
```
|
||||
ai-chat-exporter projects
|
||||
```
|
||||
|
||||
It lists every project your conversations actually belong to, marks which are
|
||||
missing from `.env`, and prints a paste-ready `CHATGPT_PROJECT_IDS=` line
|
||||
(`--write` updates `.env` for you). If your ChatGPT account does not include
|
||||
the project on conversation summaries, add `--deep` and it reads each
|
||||
conversation's detail instead — slower, one request per conversation, but
|
||||
complete.
|
||||
|
||||
Why it matters: project attribution is resolved from each conversation's own
|
||||
`gizmo_id`, so exports file correctly whether or not a project is configured.
|
||||
But the *listing* pass still needs `CHATGPT_PROJECT_IDS` — conversations that
|
||||
live only inside a project never appear in the default conversation list, so an
|
||||
unlisted project's chats are never fetched at all. Every export run also names
|
||||
any unconfigured project it encounters.
|
||||
|
||||
To find them by hand instead:
|
||||
|
||||
1. Open ChatGPT and click a Project in the left sidebar
|
||||
2. Look at the browser URL — it will look like:
|
||||
`https://chatgpt.com/g/g-p-68c2b2b3037c8191890036fb4ae3ed9f-my-project/project`
|
||||
3. Copy the `g-p-…` part (everything up to but not including the slug after the second `-`)
|
||||
|
||||
Add all your project IDs to `.env` as a comma-separated list:
|
||||
|
||||
```
|
||||
CHATGPT_PROJECT_IDS=g-p-68c2b2b3037c8191890036fb4ae3ed9f,g-p-anotherprojectid
|
||||
```
|
||||
|
||||
The `auth` wizard can also guide you through this step interactively.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Sessions
|
||||
|
||||
The `claude-code` provider archives your local [Claude Code](https://claude.com/claude-code) agent transcripts — no tokens, no API, no ToS exposure. Sessions are read from `~/.claude/projects/`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider claude-code
|
||||
ai-chat-exporter joplin --provider claude-code
|
||||
```
|
||||
|
||||
Exports are prose-only by default: your prompts and Claude's write-ups are kept, tool activity is grouped into one-line placeholders (`> 🔧 Tool output — 14 calls: Read ×9, Bash ×2 (86KB) — omitted`), and internal reasoning is dropped (counted in the run summary). Set `EXPORTER_HIDDEN_CONTENT=full` to keep everything. Sessions become notes under their own top-level **`AI-ClaudeCode`** Joplin notebook, in a sub-notebook per launch folder. The provider is included in `--provider all` whenever a sessions directory exists.
|
||||
|
||||
**Subagents.** Claude Code stores subagent (Task-tool) transcripts as separate files under `<session>/subagents/`; each is folded into its parent session inline, as a collapsible `<details>` block labeled with the subagent's type and description (its own tool traffic is collapsed like the main dialogue).
|
||||
|
||||
**Repo tags.** Sessions launched from a workspace root all share one folder-named notebook, so titles carry the repos each session touched — `Resume StartWRT project work [start-technologies]` — for at-a-glance scanning and search. A file's repo is the git repository it lives in (nearest ancestor with a `.git`), resolved from the tool paths in the transcript, so work is tagged wherever it happened — even across workspaces — and config/one-off files are ignored (they aren't repos). Repos are frequency-ordered and capped at 3. To never tag specific repos, set `CLAUDE_CODE_REPO_TAG_IGNORE` (comma-separated). Note this reads your current git layout, so a repo you later delete or move drops from the tag on re-export.
|
||||
|
||||
**Multiple locations.** By default the provider scans `~/.claude/projects/` plus `$CLAUDE_CONFIG_DIR/projects` when `CLAUDE_CONFIG_DIR` is set. To scan additional roots (e.g. other machines' sessions copied onto this box), set `CLAUDE_CODE_DIR` to a `:`-separated list:
|
||||
|
||||
```bash
|
||||
CLAUDE_CODE_DIR="$HOME/.claude/projects:/mnt/backup/laptop/.claude/projects"
|
||||
```
|
||||
|
||||
Sessions from all roots are merged by folder (no per-machine label); if the same session UUID appears in two roots, the newer copy wins. Note the exporter only sees this machine's disk and cannot recover sessions Claude Code has already pruned — run it regularly.
|
||||
|
||||
---
|
||||
|
||||
## Codex Sessions
|
||||
|
||||
The `codex` provider archives your local [Codex CLI](https://chatgpt.com/codex) agent transcripts — same deal as Claude Code: no tokens, no API, no ToS exposure. Rollout files are read from `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`.
|
||||
|
||||
```bash
|
||||
ai-chat-exporter export --provider codex
|
||||
ai-chat-exporter joplin --provider codex
|
||||
```
|
||||
|
||||
Exports are prose-only by default, with the same placeholder format as Claude Code, into their own top-level **`AI-Codex`** Joplin notebook. Repo tags work the same way (`CODEX_REPO_TAG_IGNORE` to suppress), and additional roots can be scanned with `CODEX_DIR` (`:`-separated); `CODEX_HOME`'s `sessions/` is picked up automatically when that variable is set.
|
||||
|
||||
Three things differ from Claude Code, all forced by how Codex stores its data:
|
||||
|
||||
**Reasoning cannot be exported.** Codex encrypts it at rest — every reasoning record carries `encrypted_content` with no plaintext summary in any layer of the file. It is always dropped and counted; `EXPORTER_HIDDEN_CONTENT=full` cannot bring it back.
|
||||
|
||||
**Incomplete tool calls are reported.** Codex writes each session twice in one file: its own typed items (what actually ran) and the raw model-facing wire format (everything attempted). The provider reads the typed layer — it is already decoded, and it omits harness plumbing that would otherwise need stripping — but cross-checks the raw layer for calls that never produced a result, so placeholders read `3 calls: exec_command ×3 (+2 did not complete)`. Those are commands that failed to launch, that you aborted, or that were still running when the turn ended.
|
||||
|
||||
**Cloud tasks are out of scope.** `codex cloud` tasks run server-side and are reachable at `chatgpt.com/backend-api/api/codex/tasks`, but local CLI sessions are never uploaded there, so the cloud API is not an alternative source for these transcripts and this provider stays entirely offline. If you start using `codex cloud exec`, those transcripts would be cloud-only and would need separate work.
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
# Running on a Schedule
|
||||
|
||||
Installing the daily run on Linux and Windows, what to know before relying on
|
||||
it, and how results reach your phone.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [CLI Reference](cli.md) · [Troubleshooting](troubleshooting.md)
|
||||
|
||||
---
|
||||
|
||||
## Scheduling a Daily Run
|
||||
|
||||
`sync` chains `export` then `joplin` in one invocation, which is what a scheduler
|
||||
wants — one command, and a meaningful exit code so a failed run is visible
|
||||
instead of silent.
|
||||
|
||||
```bash
|
||||
./ai-chat-exporter sync # every configured provider
|
||||
./ai-chat-exporter sync --provider codex # just one
|
||||
```
|
||||
|
||||
### Linux (systemd user timer)
|
||||
|
||||
```bash
|
||||
./scheduling/install-systemd-timer.sh --provider claude-code --provider codex
|
||||
./scheduling/install-systemd-timer.sh --uninstall
|
||||
```
|
||||
|
||||
Defaults to 09:00 daily (`--time 21:30` to change). `Persistent=true` means a
|
||||
machine that was off at the scheduled time runs the archive at next boot rather
|
||||
than skipping the day. To archive while logged out, `loginctl enable-linger $USER`.
|
||||
|
||||
Check on it with `systemctl --user list-timers aichat-sync.timer` and
|
||||
`journalctl --user -u aichat-sync.service -n 50`.
|
||||
|
||||
### Windows (Task Scheduler)
|
||||
|
||||
```powershell
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Provider chatgpt,claude
|
||||
.\scheduling\Register-AiChatSyncTask.ps1 -Unregister
|
||||
```
|
||||
|
||||
Per-user task, no admin rights needed. `-StartWhenAvailable` is the counterpart
|
||||
of systemd's `Persistent=true`. The task runs `scheduling\run-sync.ps1` with its
|
||||
window hidden; Task Scheduler keeps no output, so each run's output is appended
|
||||
to `cache\logs\scheduled-sync.log` (rolled over to `.1` past 1 MB).
|
||||
|
||||
Check on it with `Get-ScheduledTaskInfo -TaskName AiChatExporterSync` (look at
|
||||
`LastTaskResult`: 0 is success) and
|
||||
`Get-Content cache\logs\scheduled-sync.log -Tail 50`.
|
||||
|
||||
### What to know before you rely on it
|
||||
|
||||
**List only the providers that work unattended on that machine.** The local
|
||||
providers (`claude-code`, `codex`) need no credentials and always work. The web
|
||||
providers depend on a session token that expires and can only be refreshed by
|
||||
hand via DevTools — so on a machine where that token is stale, scheduling them
|
||||
means a failed run every single day, which is a good way to learn to ignore
|
||||
failures you actually want to see. That is why `--provider` is repeatable in both
|
||||
installers: schedule the coding machine for `claude-code` + `codex`, the browser
|
||||
machine for `chatgpt` + `claude`.
|
||||
|
||||
**Both installers pass `--joplin-optional`.** If Joplin desktop isn't running,
|
||||
the sync warns instead of failing: the export has already captured the local
|
||||
transcripts (the part that can disappear), and the notes are rebuilt from the
|
||||
cache on the next run that finds Joplin up.
|
||||
|
||||
**One provider failing does not skip the rest.** Both installers loop over the
|
||||
providers in a single action rather than one action each, because systemd
|
||||
`oneshot` stops at the first failing `ExecStart` and Task Scheduler reports only
|
||||
the last action's result. Every provider is attempted; the run still exits
|
||||
non-zero if any failed. The loop is `scheduling/run-sync.sh` on Linux (the
|
||||
unit's `ExecStart`) and `scheduling\run-sync.ps1` on Windows (the task's action).
|
||||
|
||||
### Getting notified
|
||||
|
||||
A scheduled run is silent by default. Output goes to three pull-only places: the
|
||||
exporter's own log (`cache/logs/exporter.log`), the systemd journal on Linux
|
||||
(`journalctl --user -u aichat-sync.service`), and on Windows Task Scheduler's
|
||||
`LastTaskResult` plus `cache\logs\scheduled-sync.log`.
|
||||
|
||||
To have runs report back, set an [ntfy](https://ntfy.sh) topic in `.env`:
|
||||
|
||||
```bash
|
||||
NTFY_TOPIC=my-archive-topic
|
||||
```
|
||||
|
||||
Then check it works before waiting on a scheduled run:
|
||||
|
||||
```bash
|
||||
ai-chat-exporter notify # show settings
|
||||
ai-chat-exporter notify --test # send a test push
|
||||
```
|
||||
|
||||
`sync` pushes a result whenever a topic is configured — a success carries the
|
||||
per-provider counts at low/default priority, a failure carries the reason at
|
||||
high priority with an alert tag, so a failed archive is distinguishable from a
|
||||
quiet one on your phone. `NTFY_NOTIFY=failure` notifies only on failure; `off`
|
||||
disables it; `--notify` / `--no-notify` override per run.
|
||||
|
||||
`sync` can only push from the end of a run it finished. A crash, an exit before
|
||||
the sync starts (the ToS gate, a cache error) or a launcher that can't build its
|
||||
venv sends nothing — and since each provider pushes separately, the ones that
|
||||
succeeded still say "OK", so a dead provider looks like a quiet day. The
|
||||
scheduler wrappers (`scheduling/run-sync.sh`, `scheduling\run-sync.ps1`) close
|
||||
that gap: any run that exits non-zero without the
|
||||
app having reported it gets a high-priority **FAILED** push naming the provider
|
||||
and, for a crash, the exception's class (`claude-code: crashed (RecursionError)`)
|
||||
— the class only, never its message, which can carry a conversation title. The
|
||||
traceback is in the journal on Linux and in `cache\logs\scheduled-sync.log` on
|
||||
Windows.
|
||||
|
||||
The message includes the **machine name**, which matters because both machines
|
||||
archive into one topic. It contains counts only — never conversation titles. A
|
||||
topic on public ntfy.sh is readable by anyone who knows its name, so if you want
|
||||
it private, self-host (`NTFY_SERVER`) or use an access-controlled topic with
|
||||
`NTFY_TOKEN`.
|
||||
|
||||
A notification is never fatal: if ntfy is unreachable, the run logs a warning and
|
||||
still reports its real exit code.
|
||||
|
||||
**Acknowledge the ToS notice once, interactively.** It's stored in the cache
|
||||
manifest per machine. Until then a scheduled run exits 1 with an explanation
|
||||
rather than hanging on a prompt no one can answer.
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
# Troubleshooting
|
||||
|
||||
What the cache does, what each failure means, and how to file a bug report.
|
||||
|
||||
← [README](../README.md) · [Providers and Tokens](providers.md) · [CLI Reference](cli.md) · [Scheduling](scheduling.md)
|
||||
|
||||
---
|
||||
|
||||
## How the Cache Works
|
||||
|
||||
The cache manifest lives at `cache/manifest.json` (inside the install directory) and records every exported conversation: its title, project, `updated_at` timestamp, output file path, and (after Joplin sync) the Joplin note ID.
|
||||
|
||||
On every `export` run:
|
||||
1. Fetch the full conversation list from the provider
|
||||
2. Compare each conversation's `updated_at` against the manifest
|
||||
3. Export only conversations that are new or have been updated
|
||||
4. Write each successfully exported conversation to the manifest **immediately** (not batched)
|
||||
|
||||
On every `joplin` run:
|
||||
1. Read the manifest to find conversations not yet synced to Joplin, or re-exported since last sync
|
||||
2. Push each pending Markdown file to Joplin (create or update)
|
||||
3. Store the Joplin note ID in the manifest so subsequent runs update rather than duplicate
|
||||
|
||||
**This design makes every run inherently resumable.** If the tool is interrupted for any reason — rate limit, network drop, Ctrl+C, crash — simply re-run the same command. It will skip already-processed conversations and continue from where it stopped.
|
||||
|
||||
A conversation whose assets failed transiently also carries `pending_media` (the
|
||||
asset refs still missing) and `pending_media_attempts`. While `pending_media` is
|
||||
set and the attempt cap is not reached, step 2 treats the conversation as stale
|
||||
so the next run re-fetches those assets — see
|
||||
[Embedded images and media](#embedded-images-and-media). A run that recovers them
|
||||
clears the key.
|
||||
|
||||
To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
|
||||
|
||||
To re-export a single conversation, delete its entry from `cache/manifest.json`
|
||||
and re-run. (There is deliberately no `cache --reset --conversation` flag; see
|
||||
`FUTURE.md`.)
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `Authentication failed` (401, or 403 from Claude)
|
||||
Your session token has expired.
|
||||
- Run `ai-chat-exporter auth` to get a new token interactively
|
||||
- Or manually copy a fresh cookie value into your `.env` file
|
||||
|
||||
Note: neither token's expiry can be read client-side. Claude's `sessionKey` is an opaque string, and claude.ai reports an invalid one as **403** "Invalid authorization" (`account_session_invalid`), not 401. ChatGPT's token is an encrypted JWE; `doctor` reads the `error` field of `/api/auth/session` instead. See [When Tokens Expire](providers.md#when-tokens-expire).
|
||||
|
||||
### `429 Rate Limited`
|
||||
The tool automatically pauses, saves progress, and exits with a clear message showing how many conversations were exported vs remaining. Just re-run the same export command to resume — the cache picks up exactly where it left off.
|
||||
|
||||
### `HTTP 500` / `503` on an asset download
|
||||
The provider's asset storage, not your setup. Retries and the cross-run recovery
|
||||
described in [Embedded images and media](#embedded-images-and-media) handle this;
|
||||
the run reports `media failed` and the next run re-fetches. Measured 2026-10-20:
|
||||
two assets lost this way to a 500 and a 503 both downloaded cleanly an hour later.
|
||||
|
||||
### Joplin: "JOPLIN_API_TOKEN is not set"
|
||||
You need to configure the token before running the `joplin` command:
|
||||
1. Open Joplin desktop
|
||||
2. Go to Tools → Options → Web Clipper
|
||||
3. Enable the Web Clipper service
|
||||
4. Copy the Authorization token shown on that page
|
||||
5. Add `JOPLIN_API_TOKEN=<token>` to your `.env` file
|
||||
|
||||
### Joplin: "Joplin is not responding"
|
||||
Joplin desktop must be running when you run the `joplin` command. The Web Clipper service shuts down when Joplin is closed.
|
||||
|
||||
### Joplin: "Joplin rejected the API token (HTTP 401)"
|
||||
The token in `JOPLIN_API_TOKEN` doesn't match what Joplin expects. Get a fresh token from Joplin → Tools → Options → Web Clipper → Authorization token.
|
||||
|
||||
### Joplin: note timed out
|
||||
If you see a timeout error, Joplin took longer than `JOPLIN_REQUEST_TIMEOUT` seconds (default: 30) to respond. Possible causes:
|
||||
- The conversation is very large and Joplin is slow to index it
|
||||
- Joplin is busy syncing or loading a large library
|
||||
- Joplin has frozen — try restarting it
|
||||
|
||||
To increase the timeout: add `JOPLIN_REQUEST_TIMEOUT=60` to your `.env`.
|
||||
|
||||
### ChatGPT project conversations not appearing
|
||||
Make sure you've added the project IDs to `CHATGPT_PROJECT_IDS` in your `.env`. See [ChatGPT Projects](providers.md#chatgpt-projects) for how to find them. Project conversations are not included in the default conversation listing — they must be fetched separately.
|
||||
|
||||
### Schema warnings in logs (`Unexpected API response shape`)
|
||||
The provider's internal API may have changed. Run with `--debug`, sanitize the output (remove any personal content), and check the project's GitHub Issues for known fixes.
|
||||
|
||||
### Non-text content warnings
|
||||
Since v0.4.0, rich content is preserved as typed blocks in the export. ChatGPT voice transcripts render as text and audio assets as `📎 File attached` placeholders with size and duration metadata. Anything the extractor doesn't recognise renders as a visible `> ⚠️ Unsupported content` block naming the type and observed keys, *and* increments a counter in the post-export summary so you can tell whether real content is being silently skipped.
|
||||
|
||||
### Embedded images and media
|
||||
Since v0.6.0, image attachments are downloaded by default into a `media/` folder beside each export and inlined as ``. Audio and other files download only with `EXPORTER_DOWNLOAD_MEDIA=all`. On the next `joplin` run these are uploaded as Joplin resources so they render inside the note. Set `EXPORTER_DOWNLOAD_MEDIA=off` to skip downloads entirely.
|
||||
|
||||
A failed asset **never fails the export** — the conversation is archived with its
|
||||
text placeholder, and the failure is tallied as `media failed` in the run summary
|
||||
and in the push notification. What happens next depends on *why* it failed:
|
||||
|
||||
| Bucket | Meaning | Retried? |
|
||||
|---|---|---|
|
||||
| `expired-or-missing` | 404 — the upload was deleted server-side | No. The bytes are gone. |
|
||||
| `forbidden` | 403 — metadata survives, bytes do not (see the 2026-08-17 investigation in `CHANGELOG.md`) | No. |
|
||||
| `download-error` | 5xx, a timeout, a network blip | **Yes** — within the run, then on up to 3 later runs |
|
||||
| `skipped-provider-unwell` | the run's circuit breaker was open | **Yes**, on a later run |
|
||||
| `unparseable-ref` / `write-error` | local problem, not the provider's | No |
|
||||
|
||||
**Within a run**, an asset download is two requests — one for a signed URL, one
|
||||
for the bytes — and each is retried on a 5xx: immediately once, then after about
|
||||
2, 4, 8 and 16 seconds with jitter, honouring `Retry-After` when the server sends
|
||||
one. Retries for a single asset are capped by a wall-clock budget so a hanging
|
||||
endpoint cannot consume the run's whole execution window.
|
||||
|
||||
**Across runs**, an asset that failed transiently is recorded against its
|
||||
conversation in the cache manifest, and the next run re-exports that conversation
|
||||
to re-fetch it. Only the missing asset is fetched — media already on disk is
|
||||
reused, and Joplin resource IDs are keyed by path, so the re-sync uploads just
|
||||
the new file. After 3 unsuccessful runs the asset is treated as lost and the
|
||||
conversation stops being re-exported.
|
||||
|
||||
**Circuit breaker.** If 5 transient failures land back to back, asset downloads
|
||||
stop for the rest of the run and the remaining assets are recorded for a later
|
||||
attempt. A provider's storage having a bad day should cost one quiet morning, not
|
||||
a run that overruns its two-hour scheduled window. Permanent failures never count
|
||||
toward this — a deleted upload says nothing about the provider's health.
|
||||
|
||||
### Tool output / hidden context collapsed
|
||||
Since v0.6.0, content that was invisible in the ChatGPT web UI is collapsed by default to one-line placeholders like `> 🔧 Tool output — file_search (15.0 KB) — omitted`. This is ChatGPT's file-retrieval tool re-injecting your attached files on every run — it can be 90% of a conversation's bytes while containing none of the dialogue. Custom Instructions collapse the same way (`> ℹ️ Hidden context`). The post-export summary lists every collapsed message by origin with the total KB omitted. To keep everything, set `EXPORTER_HIDDEN_CONTENT=full` in `.env` or pass `--hidden-content full`.
|
||||
|
||||
### Empty export / all conversations skipped
|
||||
No new or updated conversations since your last run. To verify: `ai-chat-exporter cache --show`. To force a full re-export: `ai-chat-exporter cache --clear`.
|
||||
|
||||
### Filing a bug report
|
||||
1. Run with `--debug`: `ai-chat-exporter export --debug 2>&1 | tee debug.log`
|
||||
2. Remove any personal conversation content from `debug.log`
|
||||
3. Open a GitHub Issue with the sanitized log and the exact command you ran
|
||||
|
||||
Reference in New Issue
Block a user