diff --git a/CHANGELOG.md b/CHANGELOG.md index f2cfef4..c22c965 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [Unreleased] ### Fixed +- **The terms-of-service gate exited 0 without a terminal.** `click.prompt` raises `Abort` on a closed stdin, which the handler treated as a user Ctrl-C and exited 0 — so a scheduled run on a machine that had never acknowledged the notice would report success having archived nothing. Non-interactive invocations now exit 1 with an explanation of how to clear the gate once by hand. Found by running the new systemd unit rather than by reading the code. - **A single U+0085 in a transcript silently dropped a whole record.** Both local providers split session files with `str.splitlines()`, which breaks not just on `\n` but on U+0085 (NEL), U+2028 and U+2029 — all of which are legal *inside* a JSON string and are written literally by Codex (Rust does not escape non-ASCII). One NEL in captured command output shredded one record into unparseable fragments; the parser logged "skipped 3 unparseable line(s)" and lost the record. Found while exporting a real rollout. Both providers now split on `\n` only, and both have regression tests that write their fixtures with `ensure_ascii=False` — with `json.dumps`' default the hazardous characters are escaped and the bug cannot reproduce. - **Deleted uploads are no longer reported as permission errors.** ChatGPT's `/backend-api/files/{id}/download` answers a *missing* asset with `403 {"detail":"Forbidden"}`, which reads like an auth failure and isn't one. Measured live 2026-08-17 across 18 such assets: every one returned `404 {"detail":"File not found"}` on `/files/{id}`, while assets that downloaded fine returned 200 on both in the same session, and `ChatGPT-Account-Id` made no difference. A 403 is now confirmed against the metadata endpoint before being reported (one extra request on the failure path only, none on success) and a confirmed-missing asset is logged as gone and counted as `expired-or-missing`. A 403 on an asset that *does* still exist is left alone as `forbidden` — that one would be a real problem. - **4xx errors now report why.** `_make_request` ended non-retryable statuses with `raise_for_status()`, whose curl_cffi message is `HTTP Error {code}: {reason}` — and HTTP/2 carries no reason phrase, so a refused request logged as bare `HTTP Error 403:` and the response body (the only explanation the provider gives) was discarded. The body's `detail`/`error`/`message` is now carried into the `ProviderError`, redacted and truncated. This is what made the media 403s on `GET /backend-api/files/{id}/download` undiagnosable. @@ -13,6 +14,9 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). - **`tests/test_config.py::TestSessionLimiterConfig::test_defaults` depended on the developer's `.env`.** `load_config()` calls `load_dotenv(override=False)`, which re-populated the variable the test had just deleted — so it passed only on a machine with no `.env`. The test now stubs dotenv discovery. ### Added +- **`ai-chat-exporter` / `ai-chat-exporter.cmd` launchers — no virtualenv ceremony.** `cd` into the repo and run; the wrapper creates `.venv`, installs dependencies on first run, and reinstalls when `pyproject.toml` changes. A fresh clone goes from nothing to a working command in one step (measured: ~10s), on Linux/macOS and Windows alike, which matters for a tool meant to run on several machines. `cmd.exe` searches the current directory before `PATH`, so Windows needs no `.\` prefix. The working directory is deliberately not changed — `.env`, `cache/` and `exports/` still resolve against it, which is what lets one checkout archive different machines into different places — but the wrapper now warns when you run it from elsewhere, because a different `cache/manifest.json` silently starts a *second* archive rather than failing. +- **`sync` command — `export` then `joplin` in one invocation, with a real exit code.** Intended for schedulers (and the "trivial add-on" FUTURE.md §7 anticipated): it exits non-zero if any conversation failed to export or any note failed to sync, so a scheduled run that achieved nothing is distinguishable from one that had nothing to do. `--skip-joplin` exports only; `--joplin-optional` downgrades an unreachable Joplin to a warning, since the export has already captured the local transcripts and the notes rebuild from the cache on the next run that finds Joplin up. +- **Daily scheduling for both platforms.** `scheduling/install-systemd-timer.sh` (systemd user timer, `Persistent=true` so a machine that was off catches up at boot) and `scheduling/Register-AiChatSyncTask.ps1` (per-user Task Scheduler entry, `-StartWhenAvailable`). `--provider` is repeatable in both, because the right set differs per machine: the local providers need no credentials and always work unattended, while a web provider whose session token has expired would fail the job every single day and train you to ignore it. - **Codex CLI provider (`--provider codex`).** Archives local Codex agent transcripts from `~/.codex/sessions/**/rollout-*.jsonl` — local-only, like `claude-code`: no tokens, no rate limits, no ToS exposure. Sessions land in their own top-level `AI-Codex` Joplin notebook, with the same prose-only default, repo tags (`CODEX_REPO_TAG_IGNORE`) and multi-root scanning (`CODEX_DIR`, plus `$CODEX_HOME/sessions`). Codex writes each session twice in one file and the choice between the two layers is the whole design. `response_item` records are the model-facing wire format, where a tool call arrives as *JavaScript* (`tools.exec_command({...})`) because Codex's `exec` tool is code-mode; `event_msg`/`item_completed` records are Codex's own typed items, already decoded into `CommandExecution`/`FileChange`/`Extension` with argv, cwd, exit code and output as fields. Measured over 7 sessions on 0.147.0 (2026-08-18), the typed layer is 1:1 with the raw layer for prose (91 `AgentMessage` ↔ 91 assistant messages, sharing ids) and additionally omits every piece of harness plumbing — all 51 `developer`-role messages plus the 7 `# AGENTS.md instructions…` and 1 `` injections — which the Claude Code provider has to strip by regex. So the typed layer is parsed for content. diff --git a/FUTURE.md b/FUTURE.md index 8aba5fb..1353bd2 100644 --- a/FUTURE.md +++ b/FUTURE.md @@ -312,6 +312,40 @@ session-token freshness without a browser. There is no headless context to keep fresh; the weekly manual DevTools refresh is acceptable. (Local cookie extraction remains a dead end regardless — see #2.) +### REOPENED as a TODO (2026-08-18) — centralization, not durability + +Worth revisiting, for a reason the 2026-06-28 decision did not weigh. That +decision rested on "the source conversations live in the providers' clouds and +can be re-downloaded". **That is no longer true of half the providers.** +`claude-code` (shipped v0.6.0) and `codex` (shipped 2026-08-18) read transcripts +that exist *only* on the machine that produced them — Codex prunes its rollout +files, and neither is recoverable from any cloud. The re-download premise now +covers the web providers only. + +The new motivation is consolidation rather than durability: work is split across +machines — coding sessions (`claude-code`, `codex`) on the Linux box, web chats +(`chatgpt`, `claude`) on the Windows box — and each archives to its own local +`exports/` + Joplin. A StartOS service would give one server-side corpus of all +conversations from everywhere, instead of per-machine islands that only meet +inside Joplin. + +What this would need, and what it would *not*: + +- **Not** a headless web-provider login. The hard sub-problem the original drop + retired stays retired: the web providers can keep running interactively on the + machine that has the browser, pushing their output to the server. Only the + local providers need to run server-side, and they need no tokens at all. +- Machines would need to reach the server — a sync/push step, or the service + reading transcript directories exported from each machine. +- Per-machine identity in the corpus, which the exporter currently does not + track: `claude_code.resolve_roots` deliberately merges multiple roots with "no + per-machine label". Centralizing would make that label load-bearing. +- Conflict handling for one conversation seen by two machines. The cache + manifest is per-machine today. + +Meanwhile Joplin is sufficient — it already syncs (encrypted) offsite, and it is +where the archive is actually read. This is a "nice eventually", not a gap. + ## 9. Token Validity on `doctor` — IMPLEMENTED (2026-06-28) Shipped: `doctor` now adds a "ChatGPT token active" check via diff --git a/README.md b/README.md index c1d4b01..8ad581c 100644 --- a/README.md +++ b/README.md @@ -32,10 +32,31 @@ This tool is designed for a single user backing up their own conversations. Do n ```bash git clone -cd ai-chat-exporter -python3 -m venv .venv -source .venv/bin/activate -pip install -e ".[dev]" +cd AIChatExporter +./ai-chat-exporter doctor +``` + +That's the whole install. The `ai-chat-exporter` wrapper creates `.venv` and +installs dependencies on first run, and reinstalls whenever `pyproject.toml` +changes — so there is no `python3 -m venv` / `source .venv/bin/activate` to +remember, on this machine or the next one you clone onto. Every command in this +README works the same way: + +```bash +./ai-chat-exporter export --provider all +./ai-chat-exporter sync +``` + +The wrapper does **not** change directory: `.env`, `cache/` and `exports/` all +resolve against your current directory, which is what lets one checkout archive +different machines into different places. Run it from the repo. (It warns if you +don't, because a different working directory means a different `cache/manifest.json` +— which would re-export everything and orphan your existing Joplin notes.) + +Prefer the traditional route, or want the test dependencies? That still works: + +```bash +python3 -m venv .venv && source .venv/bin/activate && pip install -e ".[dev]" ``` ### Windows @@ -44,12 +65,28 @@ No admin access required. Run these in **Command Prompt** (`cmd.exe`) — it's t ```bat git clone -cd ai-chat-exporter -python -m venv .venv -.venv\Scripts\activate -pip install -e ".[dev]" +cd AIChatExporter +ai-chat-exporter doctor ``` +`ai-chat-exporter.cmd` does the same bootstrap as the POSIX wrapper — it creates +`.venv` and installs dependencies on first run. No `python -m venv`, no +`.venv\Scripts\activate`. + +**How you invoke it differs between the two Windows shells:** + +| Shell | Command | +|---|---| +| Command Prompt (`cmd.exe`) | `ai-chat-exporter export --provider all` | +| PowerShell | `.\ai-chat-exporter.cmd export --provider all` | + +`cmd.exe` searches the current directory before `PATH` and resolves the bare name +through `PATHEXT`, so it finds `ai-chat-exporter.cmd` with no prefix and no +extension. PowerShell deliberately does *not* search the current directory, and +`.\ai-chat-exporter` there would resolve to the extensionless POSIX script, which +PowerShell cannot execute — so name the `.cmd` explicitly. Command Prompt is the +simpler of the two here. + All `ai-chat-exporter` commands work identically in Command Prompt. **Using PowerShell instead?** If you prefer PowerShell, you may need to allow script execution first (one-time, current user only): @@ -284,6 +321,69 @@ Three things differ from Claude Code, all forced by how Codex stores its data: --- +## Scheduling a Daily Run + +`sync` chains `export` then `joplin` in one invocation, which is what a scheduler +wants — one command, and a meaningful exit code so a failed run is visible +instead of silent. + +```bash +./ai-chat-exporter sync # every configured provider +./ai-chat-exporter sync --provider codex # just one +``` + +### Linux (systemd user timer) + +```bash +./scheduling/install-systemd-timer.sh --provider claude-code --provider codex +./scheduling/install-systemd-timer.sh --uninstall +``` + +Defaults to 09:00 daily (`--time 21:30` to change). `Persistent=true` means a +machine that was off at the scheduled time runs the archive at next boot rather +than skipping the day. To archive while logged out, `loginctl enable-linger $USER`. + +Check on it with `systemctl --user list-timers aichat-sync.timer` and +`journalctl --user -u aichat-sync.service -n 50`. + +### Windows (Task Scheduler) + +```powershell +.\scheduling\Register-AiChatSyncTask.ps1 -Provider chatgpt,claude +.\scheduling\Register-AiChatSyncTask.ps1 -Unregister +``` + +Per-user task, no admin rights needed. `-StartWhenAvailable` is the counterpart +of systemd's `Persistent=true`. + +### What to know before you rely on it + +**List only the providers that work unattended on that machine.** The local +providers (`claude-code`, `codex`) need no credentials and always work. The web +providers depend on a session token that expires and can only be refreshed by +hand via DevTools — so on a machine where that token is stale, scheduling them +means a failed run every single day, which is a good way to learn to ignore +failures you actually want to see. That is why `--provider` is repeatable in both +installers: schedule the coding machine for `claude-code` + `codex`, the browser +machine for `chatgpt` + `claude`. + +**Both installers pass `--joplin-optional`.** If Joplin desktop isn't running, +the sync warns instead of failing: the export has already captured the local +transcripts (the part that can disappear), and the notes are rebuilt from the +cache on the next run that finds Joplin up. + +**One provider failing does not skip the rest.** Both installers loop over the +providers in a single action rather than one action each, because systemd +`oneshot` stops at the first failing `ExecStart` and Task Scheduler reports only +the last action's result. Every provider is attempted; the run still exits +non-zero if any failed. + +**Acknowledge the ToS notice once, interactively.** It's stored in the cache +manifest per machine. Until then a scheduled run exits 1 with an explanation +rather than hanging on a prompt no one can answer. + +--- + ## Output Structure All exported files go under `EXPORT_DIR`. The folder structure maps directly to Joplin notebooks. diff --git a/ai-chat-exporter b/ai-chat-exporter new file mode 100755 index 0000000..b0ea911 --- /dev/null +++ b/ai-chat-exporter @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# ai-chat-exporter — run the exporter without activating a virtualenv. +# +# cd /path/to/AIChatExporter +# ./ai-chat-exporter export --provider all +# +# Creates .venv and installs dependencies on first run, so a fresh clone on a +# new machine needs no `python3 -m venv` / `source .venv/bin/activate` ceremony. +# Reinstalls automatically when pyproject.toml changes. +# +# Working directory is deliberately NOT changed: EXPORT_DIR, CACHE_DIR and .env +# discovery are all relative to your current directory, which is what lets the +# same checkout archive different machines into different places. Run it from +# the repo (see the warning below) unless you mean otherwise. +# +# Windows equivalent: ai-chat-exporter.cmd (same directory). + +set -euo pipefail + +DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +VENV="$DIR/.venv" +PY="$VENV/bin/python" +STAMP="$VENV/.deps-stamp" + +# The cache manifest is read from ./cache by default. Running from somewhere +# else silently starts a *new, empty* archive rather than failing, which would +# re-export everything and orphan the existing Joplin notes. Warn, don't block — +# a deliberate second archive is a legitimate thing to want. +if [ "$PWD" != "$DIR" ] && [ -z "${AI_CHAT_EXPORTER_QUIET_CWD:-}" ]; then + echo "warning: running from $PWD, not $DIR" >&2 + echo " cache/ and exports/ resolve against the current directory," >&2 + echo " so this may start a separate archive. Set" >&2 + echo " AI_CHAT_EXPORTER_QUIET_CWD=1 to silence this." >&2 +fi + +find_python() { + for candidate in python3 python; do + if command -v "$candidate" >/dev/null 2>&1; then + # Needs >=3.11 (pyproject requires-python). + if "$candidate" -c 'import sys; sys.exit(0 if sys.version_info >= (3, 11) else 1)' 2>/dev/null; then + command -v "$candidate" + return 0 + fi + fi + done + return 1 +} + +if [ ! -x "$PY" ]; then + BOOTSTRAP_PY="$(find_python)" || { + echo "error: no python3 >= 3.11 found on PATH — install it and re-run." >&2 + exit 1 + } + echo "Creating virtualenv in $VENV …" >&2 + "$BOOTSTRAP_PY" -m venv "$VENV" + rm -f "$STAMP" +fi + +# Install (or refresh) dependencies when the venv is new or pyproject changed. +if [ ! -f "$STAMP" ] || [ "$DIR/pyproject.toml" -nt "$STAMP" ]; then + echo "Installing dependencies …" >&2 + "$PY" -m pip install --quiet --upgrade pip + "$PY" -m pip install --quiet -e "$DIR" + touch "$STAMP" +fi + +exec "$PY" -m src.main "$@" diff --git a/ai-chat-exporter.cmd b/ai-chat-exporter.cmd new file mode 100644 index 0000000..9803c1b --- /dev/null +++ b/ai-chat-exporter.cmd @@ -0,0 +1,83 @@ +@echo off +rem ai-chat-exporter.cmd - run the exporter without activating a virtualenv. +rem +rem Command Prompt (cmd.exe) - the documented way: +rem cd C:\path\to\AIChatExporter +rem ai-chat-exporter export --provider all +rem +rem cmd.exe searches the current directory before PATH and resolves the bare +rem name through PATHEXT (which includes .CMD), so no ".\" and no extension are +rem needed. The extensionless POSIX sibling is ignored: it is not in PATHEXT. +rem +rem PowerShell does NOT search the current directory, and ".\ai-chat-exporter" +rem there would resolve to the extensionless POSIX script, which PowerShell +rem cannot run. In PowerShell, name this file explicitly: +rem .\ai-chat-exporter.cmd export --provider all +rem +rem Creates .venv and installs dependencies on first run, so a fresh clone needs +rem no "python -m venv" / ".venv\Scripts\activate" ceremony. Reinstalls +rem automatically when pyproject.toml changes. +rem +rem The working directory is deliberately NOT changed - EXPORT_DIR, CACHE_DIR +rem and .env discovery are all relative to it. POSIX equivalent: ai-chat-exporter + +setlocal enabledelayedexpansion + +set "DIR=%~dp0" +if "%DIR:~-1%"=="\" set "DIR=%DIR:~0,-1%" +set "VENV=%DIR%\.venv" +set "PY=%VENV%\Scripts\python.exe" +set "STAMP=%VENV%\.deps-stamp" + +rem See the POSIX script for why this warns rather than blocks: cache\ resolves +rem against the current directory, so the wrong one silently starts a second +rem archive instead of failing. +if /i not "%CD%"=="%DIR%" if "%AI_CHAT_EXPORTER_QUIET_CWD%"=="" ( + echo warning: running from %CD%, not %DIR% 1>&2 + echo cache\ and exports\ resolve against the current directory, 1>&2 + echo so this may start a separate archive. Set 1>&2 + echo AI_CHAT_EXPORTER_QUIET_CWD=1 to silence this. 1>&2 +) + +if not exist "%PY%" ( + echo Creating virtualenv in %VENV% ... 1>&2 + rem The py launcher is the reliable way to get a specific version; fall back + rem to whatever "python" is if it is not installed. + where py >nul 2>&1 + if !errorlevel! equ 0 ( + py -3 -m venv "%VENV%" + ) else ( + python -m venv "%VENV%" + ) + if not exist "%PY%" ( + echo error: could not create a virtualenv - install Python 3.11+ from 1>&2 + echo python.org or the Microsoft Store, then re-run. 1>&2 + exit /b 1 + ) + if exist "%STAMP%" del "%STAMP%" +) + +rem Staleness check, in pure batch: %%~tF is the file's last-modified stamp, so +rem storing it and comparing strings needs no external process. The obvious +rem alternative - asking PowerShell to compare timestamps - costs ~1s of +rem interpreter startup on *every* command, which is a lot to pay to almost +rem always learn that nothing changed. +set "PYPROJ_TIME=" +for %%F in ("%DIR%\pyproject.toml") do set "PYPROJ_TIME=%%~tF" + +set "SAVED_TIME=" +if exist "%STAMP%" set /p SAVED_TIME=<"%STAMP%" + +if not "%SAVED_TIME%"=="%PYPROJ_TIME%" ( + echo Installing dependencies ... 1>&2 + "%PY%" -m pip install --quiet --upgrade pip + "%PY%" -m pip install --quiet -e "%DIR%" + if !errorlevel! neq 0 ( + echo error: dependency installation failed. 1>&2 + exit /b !errorlevel! + ) + > "%STAMP%" echo !PYPROJ_TIME! +) + +"%PY%" -m src.main %* +exit /b %errorlevel% diff --git a/scheduling/Register-AiChatSyncTask.ps1 b/scheduling/Register-AiChatSyncTask.ps1 new file mode 100644 index 0000000..5074fbc --- /dev/null +++ b/scheduling/Register-AiChatSyncTask.ps1 @@ -0,0 +1,83 @@ +<# +.SYNOPSIS + Register a Windows scheduled task that runs `ai-chat-exporter sync` daily. + +.DESCRIPTION + The Windows counterpart to install-systemd-timer.sh. Creates a per-user task + (no admin rights needed) that runs the exporter from this repository, with + the working directory set to the repo so .env, cache\ and exports\ resolve + exactly as they do for an interactive run. + + -Provider is repeatable. The CLI takes one provider per run, so each becomes + its own action, executed in order. List only the providers that work + unattended on this machine: a web provider whose session token has expired + fails the task every day, which trains you to ignore the failures you + actually want to notice. + +.EXAMPLE + .\scheduling\Register-AiChatSyncTask.ps1 -Provider chatgpt,claude + +.EXAMPLE + .\scheduling\Register-AiChatSyncTask.ps1 -Provider all -Time 21:30 + +.EXAMPLE + .\scheduling\Register-AiChatSyncTask.ps1 -Unregister +#> +[CmdletBinding()] +param( + [string[]]$Provider = @('all'), + [string]$Time = '09:00', + [string]$TaskName = 'AiChatExporterSync', + [switch]$Unregister +) + +$ErrorActionPreference = 'Stop' + +$repo = Split-Path -Parent $PSScriptRoot +$launcher = Join-Path $repo 'ai-chat-exporter.cmd' + +if ($Unregister) { + Unregister-ScheduledTask -TaskName $TaskName -Confirm:$false -ErrorAction SilentlyContinue + Write-Host "Removed scheduled task '$TaskName'." + return +} + +if (-not (Test-Path $launcher)) { + throw "Launcher not found at $launcher" +} + +# A single action looping over the providers, rather than one action each. +# Task Scheduler runs multiple actions in order but reports only the last one's +# result, so a failure in an earlier provider would be invisible. The loop keeps +# going after a failure and propagates a non-zero exit code. +$loop = ($Provider | ForEach-Object { "`"$launcher`" sync --provider $_ --joplin-optional || set RC=1" }) -join ' & ' +$actions = New-ScheduledTaskAction -Execute 'cmd.exe' ` + -Argument "/c set RC=0 & $loop & exit /b %RC%" ` + -WorkingDirectory $repo + +$trigger = New-ScheduledTaskTrigger -Daily -At $Time + +# StartWhenAvailable is the counterpart of systemd's Persistent=true: a machine +# that was asleep at the scheduled time runs the archive when it wakes, rather +# than skipping the day entirely. +$settings = New-ScheduledTaskSettingsSet ` + -StartWhenAvailable ` + -DontStopIfGoingOnBatteries ` + -AllowStartIfOnBatteries ` + -ExecutionTimeLimit (New-TimeSpan -Hours 2) + +Register-ScheduledTask -TaskName $TaskName ` + -Action $actions ` + -Trigger $trigger ` + -Settings $settings ` + -Description 'Export AI chat history and sync it to Joplin' ` + -Force | Out-Null + +Write-Host "Registered '$TaskName' - daily at $Time for: $($Provider -join ', ')" +Write-Host '' +Write-Host 'Next steps:' +Write-Host " * Run it once now: Start-ScheduledTask -TaskName $TaskName" +Write-Host " * Check the result: Get-ScheduledTaskInfo -TaskName $TaskName" +Write-Host " * Read the log: Get-Content '$repo\cache\logs\exporter.log' -Tail 50" +Write-Host ' * The terms-of-service notice must have been acknowledged' +Write-Host ' interactively once on this machine, or the task exits 1.' diff --git a/scheduling/install-systemd-timer.sh b/scheduling/install-systemd-timer.sh new file mode 100755 index 0000000..053a6e6 --- /dev/null +++ b/scheduling/install-systemd-timer.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# Install a systemd *user* timer that runs `ai-chat-exporter sync` on a schedule. +# +# ./scheduling/install-systemd-timer.sh --provider claude-code --provider codex +# ./scheduling/install-systemd-timer.sh --provider all --time 21:30 +# ./scheduling/install-systemd-timer.sh --uninstall +# +# User units (not system units) are the right scope: the archive is per-user, +# the .env holds that user's session tokens, and Joplin runs in their session. +# +# --provider is repeatable. The CLI takes one provider per run, so each one +# becomes its own ExecStart line, executed in order. Prefer listing only the +# providers that actually work unattended on this machine — a web provider with +# an expired session token will fail the unit every single day, which trains you +# to ignore the failure you actually want to see. + +set -euo pipefail + +REPO="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)" +UNIT_DIR="$HOME/.config/systemd/user" +NAME="aichat-sync" +TIME="09:00" +PROVIDERS=() +UNINSTALL=0 + +while [ $# -gt 0 ]; do + case "$1" in + --provider) PROVIDERS+=("$2"); shift 2 ;; + --time) TIME="$2"; shift 2 ;; + --name) NAME="$2"; shift 2 ;; + --uninstall) UNINSTALL=1; shift ;; + -h|--help) sed -n '2,20p' "${BASH_SOURCE[0]}"; exit 0 ;; + *) echo "unknown argument: $1" >&2; exit 1 ;; + esac +done + +if [ "$UNINSTALL" -eq 1 ]; then + systemctl --user disable --now "$NAME.timer" 2>/dev/null || true + rm -f "$UNIT_DIR/$NAME.timer" "$UNIT_DIR/$NAME.service" + systemctl --user daemon-reload + echo "Removed $NAME.timer and $NAME.service." + exit 0 +fi + +[ ${#PROVIDERS[@]} -eq 0 ] && PROVIDERS=("all") + +if [ ! -x "$REPO/ai-chat-exporter" ]; then + echo "error: $REPO/ai-chat-exporter is missing or not executable." >&2 + exit 1 +fi + +mkdir -p "$UNIT_DIR" + +# WorkingDirectory is the point of the whole unit: .env, cache/ and exports/ all +# resolve against it, so the scheduled run writes to the same archive an +# interactive run from this directory would. +{ + echo "[Unit]" + echo "Description=AI chat archive sync" + echo "Documentation=file://$REPO/README.md" + echo "After=network-online.target" + echo "Wants=network-online.target" + echo + echo "[Service]" + echo "Type=oneshot" + echo "WorkingDirectory=$REPO" + echo "Environment=AI_CHAT_EXPORTER_QUIET_CWD=1" + # One ExecStart per provider would stop at the first failure, silently + # skipping the rest — an expired ChatGPT token would mean codex never runs. + # Loop instead, so every provider is attempted and the unit still reports + # failure if any of them failed. + printf 'ExecStart=/bin/sh -c '\''rc=0; for p in %s; do "$0" sync --provider "$p" --joplin-optional || rc=1; done; exit $rc'\'' %s\n' \ + "${PROVIDERS[*]}" "$REPO/ai-chat-exporter" +} > "$UNIT_DIR/$NAME.service" + +# Persistent=true runs a missed schedule at the next boot — the machine being +# off at 09:00 should delay the archive, not skip it. RandomizedDelaySec keeps +# the web providers from being hit at exactly the same second every day. +cat > "$UNIT_DIR/$NAME.timer" < None: console.print(table) +# ────────────────────────────────────────────────────────────────────────────── +# sync command +# ────────────────────────────────────────────────────────────────────────────── + + +@cli.command() +@click.option( + "--provider", + type=click.Choice(["chatgpt", "claude", "claude-code", "codex", "all"], case_sensitive=False), + default="all", + show_default=True, + help="Which provider(s) to export and sync.", +) +@click.option("--since", default=None, help="Only export conversations updated after this date (YYYY-MM-DD).") +@click.option( + "--hidden-content", + type=click.Choice(["full", "placeholder", "omit"], case_sensitive=False), + default=None, + help="Overrides EXPORTER_HIDDEN_CONTENT for this run.", +) +@click.option( + "--max-conversations", + type=click.IntRange(min=1), + default=None, + help="Cap how many conversations are downloaded this run (per provider).", +) +@click.option("--skip-joplin", is_flag=True, help="Export only; do not sync to Joplin.") +@click.option( + "--joplin-optional", + is_flag=True, + help=( + "Treat an unreachable Joplin as a warning rather than a failure. For " + "scheduled runs, where Joplin desktop may simply not be open yet." + ), +) +@click.option("--dry-run", is_flag=True, help="Show what would happen without writing or sending anything.") +@click.pass_context +def sync( + ctx: click.Context, + provider: str, + since: str | None, + hidden_content: str | None, + max_conversations: int | None, + skip_joplin: bool, + joplin_optional: bool, + dry_run: bool, +) -> None: + """Export, then sync to Joplin — the whole archive run in one command. + + Equivalent to `export` followed by `joplin` with the same --provider, which + is what a scheduled run wants: one line in a systemd timer, cron entry or + Windows scheduled task. + + Unlike the individual commands, this one sets a meaningful exit code: it + exits non-zero if any conversation failed to export or any note failed to + sync. A provider whose listing call fails outright — an expired web session + token being the usual cause — counts its whole batch as failed, so a + scheduler sees a real failure rather than a silent no-op. + + Note this does *not* flag a provider that is simply unconfigured, or one + that legitimately had nothing new to export; both are ordinary success. + """ + failures: list[str] = [] + + ctx.invoke( + export, + provider=provider, + since=since, + hidden_content=hidden_content, + max_conversations=max_conversations, + dry_run=dry_run, + ) + export_summary = ctx.obj.get("last_export_summary") or {} + for prov_name, counts in export_summary.items(): + if counts.get("failed"): + failures.append(f"{prov_name}: {counts['failed']} conversation(s) failed to export") + + if skip_joplin: + console.print("[dim]Skipping Joplin sync (--skip-joplin).[/dim]") + else: + try: + ctx.invoke(joplin, provider=provider, dry_run=dry_run) + except SystemExit as e: + # `joplin` exits non-zero when the desktop app isn't reachable. On a + # timer that is routine — the machine may not be unlocked yet — and + # the export (the part that captures data which can disappear) has + # already succeeded. The notes are rebuilt from the cache on the next + # run that finds Joplin up, so nothing is lost by carrying on. + if not joplin_optional or e.code in (0, None): + raise + console.print( + "[yellow]Joplin sync skipped — Joplin is not reachable " + "(--joplin-optional). Exported files are on disk; the next run " + "with Joplin open will sync them.[/yellow]" + ) + else: + joplin_summary = ctx.obj.get("last_joplin_summary") or {} + for prov_name, counts in joplin_summary.items(): + if counts.get("failed"): + failures.append(f"{prov_name}: {counts['failed']} note(s) failed to sync") + + if failures: + err_console.print("\n[red]Sync completed with failures:[/red]") + for line in failures: + err_console.print(f" [red]•[/red] {line}") + sys.exit(1) + + console.print("\n[green]Sync complete.[/green]") + + # ────────────────────────────────────────────────────────────────────────────── # list command # ────────────────────────────────────────────────────────────────────────────── @@ -1586,6 +1713,8 @@ def joplin(ctx: click.Context, provider: str, project_filter: str | None, dry_ru finally: progress.advance(task) + ctx.obj["last_joplin_summary"] = summary + if not dry_run: _print_joplin_summary(summary) diff --git a/tests/test_cli.py b/tests/test_cli.py index c0639b5..6233794 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -415,3 +415,123 @@ class TestProjectsCommand: assert result.exit_code == 0 env_text = (Path(fs) / ".env").read_text(encoding="utf-8") assert "CHATGPT_PROJECT_IDS=g-p-new" in env_text + + +# --------------------------------------------------------------------------- +# sync command + non-interactive ToS gate +# --------------------------------------------------------------------------- + + +class TestSyncCommand: + """`sync` chains export → joplin for schedulers, with a real exit code.""" + + def _cache(self, tmp_path) -> Cache: + cache = Cache(tmp_path) + cache.acknowledge_tos() + # Non-empty last_run so the first-run doctor gate stays out of the way. + cache.mark_exported("codex", "dummy", {"updated_at": "2024-01-01T00:00:00Z"}) + return cache + + def _env(self, tmp_path) -> dict: + """A real (minimal) codex session — `export` exits 1 on no providers at + all, so an empty directory would test the wrong failure.""" + import json + + day = tmp_path / "sessions" / "2026" / "08" / "17" + day.mkdir(parents=True, exist_ok=True) + sid = "01a00e3f-a309-74a3-bf32-06c2cd87faa3" + records = [ + { + "timestamp": "2026-08-17T05:44:06.666Z", + "type": "session_meta", + "payload": {"session_id": sid, "timestamp": "2026-08-17T05:44:06.666Z", + "cwd": str(tmp_path / "ws")}, + }, + { + "timestamp": "2026-08-17T05:44:09.000Z", + "type": "event_msg", + "payload": {"type": "item_completed", "item": { + "type": "UserMessage", "id": "u1", + "content": [{"type": "text", "text": "hello"}]}}, + }, + ] + (day / f"rollout-2026-08-17T01-44-06-{sid}.jsonl").write_text( + "\n".join(json.dumps(r) for r in records), encoding="utf-8" + ) + return { + "CACHE_DIR": str(tmp_path), + "EXPORT_DIR": str(tmp_path / "exports"), + "CODEX_DIR": str(tmp_path / "sessions"), + } + + def test_skip_joplin_exits_zero(self, tmp_path): + self._cache(tmp_path) + result = CliRunner(mix_stderr=True).invoke( + cli, + ["--no-log-file", "sync", "--provider", "codex", "--skip-joplin"], + env=self._env(tmp_path), + ) + assert result.exit_code == 0 + assert "Skipping Joplin sync" in result.output + assert "Sync complete" in result.output + + def test_joplin_optional_survives_unreachable_joplin(self, tmp_path): + """Joplin being closed must not fail a scheduled run — the export is done.""" + self._cache(tmp_path) + result = CliRunner(mix_stderr=True).invoke( + cli, + ["--no-log-file", "sync", "--provider", "codex", "--joplin-optional"], + env={**self._env(tmp_path), "JOPLIN_API_URL": "http://127.0.0.1:9"}, + ) + assert result.exit_code == 0 + assert "Joplin sync skipped" in result.output + + def test_unreachable_joplin_fails_without_the_flag(self, tmp_path): + self._cache(tmp_path) + result = CliRunner(mix_stderr=True).invoke( + cli, + ["--no-log-file", "sync", "--provider", "codex"], + env={**self._env(tmp_path), "JOPLIN_API_URL": "http://127.0.0.1:9"}, + ) + assert result.exit_code == 1 + + def test_export_failures_set_nonzero_exit(self, tmp_path, monkeypatch): + """A scheduler must be able to tell a real run from a silent no-op.""" + self._cache(tmp_path) + import src.main as main_mod + + real_export = main_mod.export.callback + + def fake_export(*args, **kwargs): + import click + ctx = click.get_current_context() + ctx.obj["last_export_summary"] = { + "codex": {"exported": 0, "skipped": 0, "failed": 3} + } + + monkeypatch.setattr(main_mod.export, "callback", fake_export) + try: + result = CliRunner(mix_stderr=True).invoke( + cli, + ["--no-log-file", "sync", "--provider", "codex", "--skip-joplin"], + env=self._env(tmp_path), + ) + finally: + monkeypatch.setattr(main_mod.export, "callback", real_export) + assert result.exit_code == 1 + assert "3 conversation(s) failed to export" in result.output + + +class TestNonInteractiveTosGate: + """Without a TTY the gate must fail loudly, not exit 0 having done nothing.""" + + def test_no_tty_exits_one_with_explanation(self, tmp_path, monkeypatch): + Cache(tmp_path) # fresh cache: ToS not acknowledged + monkeypatch.setattr("sys.stdin.isatty", lambda: False) + result = CliRunner(mix_stderr=True).invoke( + cli, + ["--no-log-file", "doctor"], + env={"CACHE_DIR": str(tmp_path), "EXPORT_DIR": str(tmp_path / "exports")}, + ) + assert result.exit_code == 1 + assert "no terminal to" in result.output