AIChatExporter/README.md

# AI Chat Exporter

A personal backup tool for ChatGPT and Claude conversation history. Exports your chats to Markdown files and syncs them to [Joplin](https://joplinapp.org/) as notes. Each conversation becomes a single `.md` file with YAML frontmatter, organised into folders that map directly to Joplin notebooks.

Supports incremental sync — only new or updated conversations are exported on each run. Every run is resumable: if interrupted, re-running picks up exactly where it left off.

---

## Contents

**This page** — what it is, installing it, the first run, configuration and where files land.

- [Terms of Service Warning](#terms-of-service-warning)
- [Installation](#installation)
- [First Run: Run Doctor](#first-run-run-doctor)
- [`.env` Setup](#env-setup)
- [Output Structure](#output-structure)
- [Documentation](#documentation)
- [Future Work](#future-work)
- [Security Notes](#security-notes)

**Detail lives in `docs/`:**

| Document | Covers |
|----------|--------|
| [docs/providers.md](docs/providers.md) | Session tokens and how to refresh them, the `auth` wizard, ChatGPT Projects, Claude Code sessions, Codex sessions |
| [docs/cli.md](docs/cli.md) | CLI reference — every command and flag |
| [docs/scheduling.md](docs/scheduling.md) | The daily run on Linux and Windows, and push notifications |
| [docs/troubleshooting.md](docs/troubleshooting.md) | How the cache works, what each failure means, filing a bug report |

---

## Terms of Service Warning

**Read this before using this tool.**

This tool works by accessing **unofficial, undocumented internal web API endpoints** used by the ChatGPT and Claude web apps. These endpoints are not publicly supported by OpenAI or Anthropic and are subject to change or removal without notice.

**Use of this tool may conflict with their Terms of Service:**
- OpenAI: https://openai.com/policies/terms-of-use
- Anthropic: https://www.anthropic.com/legal/consumer-terms

**By using this tool, you accept that:**
- You are using it entirely at your own risk
- Your account could potentially be suspended for automated or scripted access
- The internal APIs this tool relies on may break at any time without notice
- This tool is for **personal archival use only** — not commercial use

This tool is designed for a single user backing up their own conversations. Do not use it to scrape data at scale or for any commercial purpose.

---

## Installation

### Linux / macOS

```bash
git clone <repo-url>
cd AIChatExporter
./ai-chat-exporter doctor
```

That's the whole install. The `ai-chat-exporter` wrapper creates `.venv` and
installs dependencies on first run, and reinstalls whenever `pyproject.toml`
changes — so there is no `python3 -m venv` / `source .venv/bin/activate` to
remember, on this machine or the next one you clone onto. Every command in this
README works the same way:

```bash
./ai-chat-exporter export --provider all
./ai-chat-exporter sync
```

The wrapper does **not** change directory: `.env`, `cache/` and `exports/` all
resolve against your current directory, which is what lets one checkout archive
different machines into different places. Run it from the repo. (It warns if you
don't, because a different working directory means a different `cache/manifest.json`
— which would re-export everything and orphan your existing Joplin notes.)

Prefer the traditional route, or want the test dependencies? That still works:

```bash
python3 -m venv .venv && source .venv/bin/activate && pip install -e ".[dev]"
```

### Windows

No admin access required. Run these in **Command Prompt** (`cmd.exe`) — it's the simplest option on Windows because it doesn't have PowerShell's script execution policy restrictions.

```bat
git clone <repo-url>
cd AIChatExporter
ai-chat-exporter doctor
```

`ai-chat-exporter.cmd` does the same bootstrap as the POSIX wrapper — it creates
`.venv` and installs dependencies on first run. No `python -m venv`, no
`.venv\Scripts\activate`.

**How you invoke it differs between the two Windows shells:**

| Shell | Command |
|---|---|
| Command Prompt (`cmd.exe`) | `ai-chat-exporter export --provider all` |
| PowerShell | `.\ai-chat-exporter.cmd export --provider all` |

`cmd.exe` searches the current directory before `PATH` and resolves the bare name
through `PATHEXT`, so it finds `ai-chat-exporter.cmd` with no prefix and no
extension. PowerShell deliberately does *not* search the current directory, and
`.\ai-chat-exporter` there would resolve to the extensionless POSIX script, which
PowerShell cannot execute — so name the `.cmd` explicitly. Command Prompt is the
simpler of the two here.

All `ai-chat-exporter` commands work identically in Command Prompt.

**Using PowerShell instead?** If you prefer PowerShell, you may need to allow script execution first (one-time, current user only):

```powershell
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
```

Then activate the venv and run commands the same way.

**Prerequisites:**
- Python 3.11 or later — install from [python.org](https://www.python.org/downloads/windows/). During installation, tick **"Add Python to PATH"**.
- Git — install from [git-scm.com](https://git-scm.com/) if not already present.

**Notes:**
- The cache manifest and logs are stored in `cache\` inside the install directory — the same as on Linux.
- File permission hardening (`chmod 600`) is silently ignored on Windows — not a concern for single-user desktop use.
- Joplin Web Clipper runs on `localhost:41184` on all platforms; no configuration changes needed.

---

## First Run: Run Doctor

Before anything else, validate your setup:

```bash
ai-chat-exporter doctor
```

This checks token presence, format, token health (via the `/api/auth/session` `error` field), directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.

### Checking for API drift

The exporter reads ChatGPT's and Claude's undocumented internal web APIs, which can change shape without notice. The worst failure for a backup tool is *silent* — a response change that makes the exporter skip or mis-parse content without erroring. Run the canary to catch that early:

```bash
ai-chat-exporter canary
```

It probes one conversation per provider and checks only the fields the parser depends on (a renamed retrieval-tool author that would bypass the content-collapse, a new content type, drifted message fields, ignored attachments). `OK` means the live schema still matches; `WARN`/`ERROR` tells you exactly what changed. Worth running before a large export or on a schedule.

---

## `.env` Setup

Copy `.env.example` to `.env` and fill in your values:

```bash
cp .env.example .env
```

### Provider tokens

| Variable | Description |
|----------|-------------|
| `CHATGPT_SESSION_TOKEN` | ChatGPT session token chunk `.0` (starts with `eyJ…`) |
| `CHATGPT_SESSION_TOKEN_1` | ChatGPT session token chunk `.1` (the remainder) |
| `CHATGPT_PROJECT_IDS` | Comma-separated ChatGPT project IDs (see below) |
| `CLAUDE_SESSION_KEY` | Your Claude session key |

### Output

| Variable | Default | Description |
|----------|---------|-------------|
| `EXPORT_DIR` | `./exports` | Where to write exported Markdown files |
| `OUTPUT_STRUCTURE` | `provider/project/year` | Folder structure (see below) |
| `EXPORTER_HIDDEN_CONTENT` | `placeholder` | What to do with content invisible in the provider's web UI (file-retrieval tool dumps, Custom Instructions): `placeholder` collapses to a one-line note with size, `full` keeps everything, `omit` drops it. Dumps can be 90% of a conversation's bytes. |
| `EXPORTER_DOWNLOAD_MEDIA` | `images` | Download conversation assets into a `media/` folder beside each export and inline them: `images` (images only), `all` (also audio/voice clips), `off` (text placeholders only). Downloaded media is uploaded to Joplin as resources on the next `joplin` run. |
| `MAX_CONVERSATIONS_PER_RUN` | unlimited | Cap downloads per export run (per provider). Runs are resumable, so re-running continues where the cap stopped — useful to spread a large first export across several sessions. |
| `REQUEST_DELAY` | `1.0` | Seconds between consecutive API requests, with small random jitter, so traffic stays human-paced. Set `0` to disable. |

### Joplin

| Variable | Default | Description |
|----------|---------|-------------|
| `JOPLIN_API_TOKEN` | — | Authorization token from Joplin Web Clipper settings |
| `JOPLIN_API_URL` | `http://localhost:41184` | Joplin API URL (change only if you've customised the port) |
| `JOPLIN_REQUEST_TIMEOUT` | `30` | Seconds before an API call times out. Increase for very large conversations. |

### Notifications

| Variable | Default | Description |
|----------|---------|-------------|
| `NTFY_TOPIC` | — | [ntfy](https://ntfy.sh) topic to push run results to. Unset disables notifications entirely. |
| `NTFY_SERVER` | `https://ntfy.sh` | Point at your own host if self-hosting. |
| `NTFY_TOKEN` | — | Bearer token, for access-controlled topics. |
| `NTFY_NOTIFY` | `always` | `always` notifies on every run, `failure` only when something failed, `off` never. |

A topic on public ntfy.sh is readable by anyone who knows its name, so
notifications carry per-provider counts and a machine name only — never
conversation titles. See [Getting notified](docs/scheduling.md#getting-notified).

### Cache & logging

| Variable | Default | Description |
|----------|---------|-------------|
| `CACHE_DIR` | `./cache` | Where to store the sync manifest |
| `LOG_FILE` | `./cache/logs/exporter.log` | Log file path (`none` to disable) |
| `AI_CHAT_EXPORTER_QUIET_CWD` | — | Set to `1` to silence the launcher's warning when run from outside the repo. Read by the `ai-chat-exporter` wrapper scripts, not by Python; the scheduler installers set it, since they always set the correct working directory. |

---

## Output Structure

All exported files go under `EXPORT_DIR`. The folder structure maps directly to Joplin notebooks.

### Default: `provider/project/year`

```
exports/
├── chatgpt/
│   ├── no-project/
│   │   └── 2024/
│   │       └── 2024-03-15_my-conversation_abc12345.md
│   └── learning-python/
│       └── 2024/
│           └── 2024-03-15_async-tutorial_def67890.md
└── claude/
    ├── no-project/
    │   └── 2024/
    │       └── 2024-06-01_docker-explained_ghi11111.md
    └── startos-packaging/
        └── 2024/
            └── 2024-06-10_manifest-setup_jkl22222.md
```

### Joplin Notebook Mapping

Each provider+project combination maps to a flat Joplin notebook created automatically by the `joplin` command:

| Export folder | Joplin notebook |
|---------------|-----------------|
| `exports/chatgpt/learning-python/` | `ChatGPT - Learning Python` |
| `exports/claude/startos-packaging/` | `Claude - Startos Packaging` |
| `exports/chatgpt/no-project/` | `ChatGPT - No Project` |
| `exports/claude/no-project/` | `Claude - No Project` |

### Other `OUTPUT_STRUCTURE` options

| Value | Result |
|-------|--------|
| `provider/project/year` (default) | `exports/claude/my-project/2024/file.md` |
| `provider/project` | `exports/claude/my-project/file.md` |
| `provider/year` | `exports/claude/2024/file.md` (projects ignored) |

### Filename format

`YYYY-MM-DD_{title-slug}_{id[:8]}.md` — e.g. `2024-06-10_manifest-setup_jkl22222.md`

---

## Documentation

This page covers installing and configuring the exporter. Everything else is in
`docs/`:

- **[Providers and Tokens](docs/providers.md)** — getting a session token out of
  DevTools, what each one's lifetime is and how expiry is detected, the `auth`
  wizard, and the setup each provider needs: ChatGPT Projects (which are not in
  the default conversation listing), Claude Code sessions, Codex sessions.
- **[CLI Reference](docs/cli.md)** — `auth`, `doctor`, `export`, `list`,
  `joplin`, `sync`, `notify`, `prune`, `projects`, `canary`, `cache`, with the
  flags and examples for each.
- **[Running on a Schedule](docs/scheduling.md)** — the systemd user timer, the
  Windows scheduled task, what to know before relying on either, and how run
  results reach your phone.
- **[Troubleshooting](docs/troubleshooting.md)** — how the cache decides what to
  re-export, every error message the tool produces and what it means, media
  download failures, and what to include in a bug report.

---

## Future Work

See `FUTURE.md` for the full roadmap. Current priority:

- **A StartOS service** that centralises every machine's conversations into one
  corpus and owns the Joplin connection, so each machine only has to upload
  (`FUTURE.md` §1)

---

## Security Notes

- All exported data is stored **locally only** — nothing is sent anywhere except to your local Joplin instance
- Exported files and the cache manifest are created with `600` permissions (owner read/write only)
- `.env` is in `.gitignore` — **never commit it**
- Session tokens are never logged, printed, or included in error messages
- The Joplin API token is only ever sent to `localhost` — it never leaves your machine
- If you accidentally commit `.env`: immediately log out and back in to invalidate the token, then remove it from git history using [BFG Repo Cleaner](https://rtyley.github.io/bfg-repo-cleaner/) or `git filter-branch`