JesseMarkowitzandClaude Opus 5
04ebb8987c
The archive does what it was built to do, unattended, on both machines: four providers, incremental and resumable, Markdown plus Joplin, on a schedule, with results pushed to a phone and failures that announce themselves. The one item left on the roadmap — a StartOS service that consolidates both machines into one corpus — is an optional convenience, not a gap. Hence 1.0.0 rather than another 0.x. The code in this release shipped across b6ce636..8c3e00a: the claude.ai 403 session-invalid fix, the fork-subagent recursion fix, the FAILED-push backstops on both platforms, and asset-download retries with cross-run recovery, a circuit breaker and media counts in the notification. This commit cuts the release and pays a documentation debt. The README had reached 939 lines / 6,490 words / 48 KB / 96 headings, doing four unrelated jobs at once, and FUTURE.md asked for the split to ride along with the next substantial README edit rather than break every existing anchor for its own sake. A release is that edit. README.md is now 303 lines and keeps only what you need to install and configure the tool; the rest moved verbatim into docs/providers.md, docs/cli.md, docs/scheduling.md and docs/troubleshooting.md. No section was rewritten or dropped and every original heading still exists, so each anchor still resolves somewhere — but bookmarks into moved sections now need the docs/ path, which is the cost this deferral was protecting against. .gitignore needed the fix FUTURE.md predicted: the blanket *.md rule (exported conversations are Markdown and may be private) matched all four new pages, and git check-ignore confirmed every one would have been silently ignored. !docs/*.md is added with a comment saying why. Gitea's rendering of relative docs/ links cannot be checked before pushing, so it still needs confirming after. Also corrects two stale claims in the changelog: the Windows backstop is no longer "not yet run under Windows PowerShell 5.1 or Task Scheduler" (confirmed on winmini today, LastTaskResult 0 — only the FAILED push itself remains unexercised there), and the test count is 414, not 409. 414 tests passing. No code changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012gCWSknsWvhLnMFwa7qnGp
110 lines
7.8 KiB
Markdown
110 lines
7.8 KiB
Markdown
# Planned Future Work
|
|
|
|
> **Status 2026-10-20 (v1.0.0).** The tool archives four providers — two web (`chatgpt`,
|
|
> `claude`) and two local agent-transcript (`claude-code`, `codex`) — on a
|
|
> schedule, on Linux and Windows, reporting results by push notification.
|
|
> **One item below is genuinely planned.** Everything else has shipped or been
|
|
> decided against.
|
|
|
|
Completed work moves to the changelog; this file holds only what is *not* built
|
|
yet. Decisions not to build something are recorded at the bottom in one line
|
|
each, so they are not re-proposed — the full investigation trails behind them
|
|
are in `FUTURE-ARCHIVE.md`.
|
|
|
|
---
|
|
|
|
# Roadmap
|
|
|
|
## 1. StartOS Service — one corpus, not per-machine islands
|
|
|
|
Each machine currently archives to its own `exports/` and its own Joplin. Work
|
|
is split across boxes — coding sessions (`claude-code`, `codex`) on the Linux
|
|
machine, web chats (`chatgpt`, `claude`) on the Windows one — so there is no
|
|
single place where all conversations exist together.
|
|
|
|
This was dropped on 2026-06-28 and **reopened 2026-08-18**, because the
|
|
reasoning behind the drop has gone stale. It rested on "the source conversations
|
|
live in the providers' clouds and can be re-downloaded". That is no longer true
|
|
of half the providers: `claude-code` and `codex` transcripts exist *only* on the
|
|
machine that produced them, Codex prunes its rollout files, and neither is
|
|
recoverable from any cloud. The motivation is also different from the one
|
|
weighed then — consolidation, not durability.
|
|
|
|
**Intended split of responsibility (decided 2026-08-18).** Each machine keeps
|
|
running the exporter locally and keeps doing what only it can do: read that
|
|
machine's local transcripts, and hold the browser session for the web providers.
|
|
What changes is where the output goes. Instead of syncing to Joplin itself, a
|
|
local run **uploads its conversations to the StartOS storage area**, and the
|
|
StartOS service owns the Joplin connection for the whole corpus.
|
|
|
|
That inverts today's arrangement and removes two problems we already have:
|
|
|
|
- **The Joplin-availability race leaves the clients.** A scheduled run currently
|
|
has to find Joplin desktop open on that same machine — the 09:02 timer run on
|
|
2026-08-18 exported fine and then skipped the sync because Joplin did not
|
|
start until 09:07. A server that is always up has no such window, and
|
|
`--joplin-optional` stops being load-bearing.
|
|
- **One Joplin integration instead of N.** Notebook naming, resource upload and
|
|
note updates run once, server-side, against one manifest — rather than each
|
|
machine independently deciding what a notebook is called and racing to update
|
|
the same note.
|
|
|
|
What this needs, and what it does *not*:
|
|
|
|
- **Not** a headless web-provider login. The hard sub-problem the original drop
|
|
retired stays retired: the web providers keep running interactively on the
|
|
machine that has the browser, and push their output to the server. Only the
|
|
local providers would run server-side, and they need no tokens at all.
|
|
- An upload step in the client — the counterpart of today's `joplin` command,
|
|
pointed at the StartOS service instead of a local Joplin API. Probably a
|
|
`push` alongside `sync`, so a scheduled client run stays one line.
|
|
- Per-machine identity in the corpus, which the exporter does not track today:
|
|
`claude_code.resolve_roots` deliberately merges multiple roots with "no
|
|
per-machine label". Centralizing makes that label load-bearing.
|
|
- Conflict handling for one conversation seen by two machines, and a decision
|
|
about whether the server or the client owns the cache manifest. It is
|
|
per-machine today, and that is what makes "already up to date" mean anything.
|
|
- A story for what the client keeps locally after a successful upload. Exports
|
|
are the only copy of `claude-code` / `codex` transcripts once Codex prunes its
|
|
rollouts, so the client should keep them rather than hand them off.
|
|
|
|
Meanwhile Joplin is sufficient — it already syncs (encrypted) offsite, and it is
|
|
where the archive is actually read. This is a "nice eventually", not a gap.
|
|
|
|
---
|
|
|
|
# Decided against
|
|
|
|
Recorded so they are not re-proposed. Full reasoning and the recon behind each
|
|
is in `FUTURE-ARCHIVE.md`.
|
|
|
|
| Item | Verdict |
|
|
|------|---------|
|
|
| Brave/Chromium cookie auto-extraction | **Not viable** (2026-06-12). App-Bound Encryption in Chrome 127+/current Brave needs admin + SYSTEM impersonation that AV flags as credential theft, and fails on Brave specifically. Auth stays manual via DevTools. |
|
|
| In-app watch/polling loop | **Dropped** (2026-06-28). Scheduling belongs to the host. Delivered instead in v0.9.0 as `sync` plus a systemd timer and a Windows scheduled task. |
|
|
| Proactive token-expiry notification | **Blocked, and moot** (2026-06-28). No trustworthy client-side expiry exists: the ChatGPT token is a JWE, Claude's is opaque, and `/api/auth/session`'s `expires` is a rolling window that advances even for a dead token. `doctor`'s `error`-based health check covers the real need. |
|
|
| Official export-ZIP fallback | **Closed** (2026-06-12). ChatGPT's official export omits project data, so it is not a real fallback here. `BaseProvider` still admits a `FileProvider` if that changes. |
|
|
| Joplin `--force` flag | Closed as not needed (2026-06-28). |
|
|
| Per-conversation cache reset | Closed as not needed (2026-06-28). Workaround: edit the manifest. |
|
|
| o1/o3 reasoning subpart reclassification | Closed (2026-06-28) — never seen in real captured data. |
|
|
| Obsidian vault output | Closed as not needed (2026-06-28). The Markdown is already Obsidian-valid; only file copying would be needed. |
|
|
| Search command | Closed as not needed (2026-06-28). `grep`/`ripgrep` over `EXPORT_DIR` covers it. |
|
|
| Claude `attachments` / `files` handling | Closed (2026-06-28) — never seen in real data; the drift canary will flag it if it appears. |
|
|
| Additional web providers (Gemini, Grok, Perplexity) | Out of scope — no significant usage to archive. |
|
|
|
|
---
|
|
|
|
# Completed
|
|
|
|
Detail for each release is in `CHANGELOG.md`.
|
|
|
|
- **v0.1.0** — Core export: ChatGPT + Claude, incremental sync, Markdown + JSON output
|
|
- **v0.2.0** — Joplin import automation (`joplin` command, create/update notes, notebook auto-creation)
|
|
- **v0.4.0** — Rich content: typed message blocks, ChatGPT voice transcripts, Custom Instructions extraction, data-loss visibility via `LossReport` and visible `unknown` blocks
|
|
- **v0.5.0** — Nested Joplin notebooks, date-prefixed note titles, flat year folders
|
|
- **v0.6.0** — Collapse tool retrieval dumps & hidden context (`EXPORTER_HIDDEN_CONTENT`); Claude Code session provider; `prune` + manifest integrity; binary content downloads; session limiter and request pacing; `export --force`
|
|
- **v0.7.0** — `canary` drift detection; real ChatGPT token-health check on `doctor`; removal of the non-viable browser-cookie path
|
|
- **v0.8.0** — Claude Code coverage reopened: subagent capture as folded `<details>`, repo `[tags]` in titles, its own `AI-ClaudeCode` notebook with self-healing note moves, multi-root scanning (`CLAUDE_CODE_DIR`, `CLAUDE_CONFIG_DIR`)
|
|
- **v0.9.0** — Codex CLI provider; launcher scripts removing the virtualenv ceremony on both platforms; `sync` with a meaningful exit code; daily scheduling for Linux and Windows; ntfy push notifications; ChatGPT project attribution via `gizmo_id` and the `projects` command; a silent data-loss fix in both local providers (`splitlines` breaking on U+0085/U+2028/U+2029)
|
|
- **v1.0.0** — Reliability pass on unattended runs: asset downloads retried on 5xx with a per-asset time budget, transiently failed assets recovered on later runs, a media circuit breaker, media failures surfaced in the push notification; a timeout on an asset no longer aborts its conversation; claude.ai's 403 session-invalid recognised as an expired key; `fork` subagents no longer recurse forever; a crashed scheduled run pushes a FAILED notification on both platforms; the README split into an overview plus `docs/`
|