fix: force re-render campaign tracking, cache dir loading, Joplin link preservation. Co-Authored-By: Fable 5

CHANGELOG.md

@@ -46,6 +46,11 @@ consolidated here.
 - Session limiter: `--max-conversations N` / `MAX_CONVERSATIONS_PER_RUN` cap downloads per run (per provider); capped-out conversations are reported as "deferred" and resumed next run.
 - Polite pacing: `REQUEST_DELAY` (default 1.0s, ±25% jitter, `0` disables) between consecutive API requests, with the existing 429 backoff as the reactive net.
 
+**Re-rendering**
+- `export --force` re-exports every conversation even if cached and unchanged, so the whole archive can be re-rendered after a formatting/feature change without `cache --clear`. Runs as a tracked campaign (a stamped start time in the manifest): each run re-renders the least-recently-exported conversations, the "still to go" count shrinks toward zero, finished providers do no further work, and the campaign auto-completes (reporting "Force re-render complete"). Combines with `--max-conversations` to spread the work across runs.
+- `mark_exported` now preserves `joplin_note_id` / `joplin_synced_at` / `joplin_resources` across re-exports (refreshing `exported_at`), so a re-export followed by `joplin` updates the existing notes instead of creating duplicates.
+- Fix: `.env` is now loaded at the very start of every command, so `CACHE_DIR` and `LOG_FILE` are honored (previously the cache silently used `./cache` regardless of `CACHE_DIR`).
+
 **Archive hygiene**
 - `prune` deletes export files not referenced by the manifest (old-layout trees, no-ID orphans), with listing, confirmation, `--dry-run`, `-y`, and empty-directory sweep. Refuses to run when the manifest references no files (so `cache --clear` + `prune` can't wipe the archive). First live run removed 420 stale files (9.4 MB).
 - `doctor` verifies manifest ↔ disk integrity: every recorded `file_path` must exist on disk.

FUTURE.md

@@ -301,16 +301,12 @@ loop.
 
 Kept for reference; not on the active roadmap.
 
-## Export `--force` Flag
+## Export `--force` Flag — SHIPPED v0.6.0
 
-Add `--force` to the `export` command to re-export already-cached conversations
-without permanently clearing the entire manifest. Useful for re-generating files
-after changing the Markdown template or output structure.
-
-Implementation: pass a `force=True` flag to `cache.get_new_or_updated()`, which
-returns all conversations regardless of cache state when force is True.
-
-Current workaround: `python -m src.main cache --clear` then re-run export.
+Implemented 2026-06-12: `export --force` passes `force=True` to
+`cache.get_new_or_updated()`. Shipped alongside a `mark_exported` fix that
+preserves Joplin links across re-exports, so a forced re-render + `joplin`
+updates existing notes instead of duplicating them.
 
 ## Joplin `--force` Flag
 

README.md

@@ -329,7 +329,22 @@ ai-chat-exporter export --output /path/to/my/notes
 ai-chat-exporter export --dry-run
 ```
 
-Options: `--provider [chatgpt|claude|claude-code|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--dry-run`
+Options: `--provider [chatgpt|claude|claude-code|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--hidden-content [full|placeholder|omit]`, `--download-media [images|all|off]`, `--max-conversations N`, `--force`, `--dry-run`
+
+**Re-rendering the whole archive after an upgrade.** New formatting or features (collapse policy, media downloads) only change conversations as they're re-exported. To re-render everything you already have, use `--force` — it re-exports every conversation even if unchanged, **without** `cache --clear`, so your Joplin note links are preserved (a later `joplin` run updates the existing notes instead of duplicating them).
+
+A force re-render runs as a tracked "campaign": each run re-renders the least-recently-exported conversations, the "still to go" count shrinks each run, and finished providers do no further work. The simplest approach is one uncapped pass:
+
+```bash
+ai-chat-exporter export --force      # re-renders everything in one paced pass
+ai-chat-exporter joplin              # update notes + upload media
+```
+
+Or spread the load with `--max-conversations`, re-running until it reports "Force re-render complete":
+
+```bash
+ai-chat-exporter export --force --max-conversations 50   # repeat until complete
+```
 
 ### `list` — List conversations
 

src/cache.py

@@ -84,7 +84,13 @@ class Cache:
         if provider not in self._data:
             self._data[provider] = {}
 
-        self._data[provider][conv_id] = {
+        # Preserve the Joplin link across re-exports. Without this, re-rendering
+        # a conversation (e.g. a forced re-export to pick up new formatting)
+        # would drop joplin_note_id, and the next sync would create a duplicate
+        # note instead of updating the existing one. exported_at is refreshed,
+        # so get_joplin_pending() still flags the note for an update.
+        prev = self._data[provider].get(conv_id) or {}
+        entry = {
             "title": metadata.get("title", ""),
             "project": metadata.get("project"),
             "created_at": metadata.get("created_at", ""),
@@ -92,20 +98,56 @@ class Cache:
             "exported_at": datetime.now(tz=timezone.utc).isoformat(),
             "file_path": metadata.get("file_path", ""),
         }
+        for carried in ("joplin_note_id", "joplin_synced_at", "joplin_resources"):
+            if carried in prev:
+                entry[carried] = prev[carried]
+
+        self._data[provider][conv_id] = entry
         self._data["last_run"] = datetime.now(tz=timezone.utc).isoformat()
         self._save()
 
-    def get_new_or_updated(self, provider: str, conversations: list[dict]) -> list[dict]:
+    def get_new_or_updated(
+        self,
+        provider: str,
+        conversations: list[dict],
+        force: bool = False,
+        campaign_at: str | None = None,
+    ) -> list[dict]:
         """Filter a conversation list to only new or updated conversations.
 
         Args:
             provider: "chatgpt" or "claude"
             conversations: List of raw conversation dicts from the provider.
                            Each must have an ``id``/``uuid`` and ``updated_at``/``update_time``.
+            force: When True, return conversations to re-render regardless of
+                   cache freshness — used by ``export --force``.
+            campaign_at: In force mode, exclude conversations already re-exported
+                   at/after this ISO timestamp (the re-render campaign start), so
+                   each run advances and finished providers do nothing.
 
         Returns:
-            Subset that needs to be exported.
+            Subset that needs to be exported, ordered oldest-export-first in
+            force mode.
         """
+        if force:
+            # Re-export for a re-render campaign, ordered least-recently-exported
+            # first (never-exported sorts first). When ``campaign_at`` is given,
+            # conversations already re-exported during this campaign
+            # (exported_at >= campaign_at) are excluded — so finished providers
+            # do no work and the remaining count shrinks to zero. Each capped run
+            # re-renders the next-oldest batch. Conversations without an id drop.
+            entries = self._data.get(provider, {})
+
+            def _exported_at(conv: dict) -> str:
+                conv_id = conv.get("id") or conv.get("uuid", "")
+                entry = entries.get(conv_id) or {}
+                return entry.get("exported_at") or ""
+
+            candidates = [c for c in conversations if c.get("id") or c.get("uuid")]
+            if campaign_at is not None:
+                candidates = [c for c in candidates if _exported_at(c) < campaign_at]
+            return sorted(candidates, key=_exported_at)
+
         result = []
         for conv in conversations:
             conv_id = conv.get("id") or conv.get("uuid", "")
@@ -249,6 +291,25 @@ class Cache:
         """Return the ISO8601 timestamp of the last export run, or None."""
         return self._data.get("last_run")
 
+    def exported_at(self, provider: str, conv_id: str) -> str:
+        """Return the recorded ``exported_at`` for a conversation, or '' if absent."""
+        entry = self._data.get(provider, {}).get(conv_id)
+        return entry.get("exported_at", "") if isinstance(entry, dict) else ""
+
+    # Force re-render campaign marker (top-level scalar; skipped by stats/clear,
+    # which only act on dict-valued provider keys).
+    def get_force_campaign(self) -> str | None:
+        """Return the active force-re-render campaign start timestamp, or None."""
+        return self._data.get("force_campaign_at")
+
+    def set_force_campaign(self, started_at: str) -> None:
+        self._data["force_campaign_at"] = started_at
+        self._save()
+
+    def clear_force_campaign(self) -> None:
+        if self._data.pop("force_campaign_at", None) is not None:
+            self._save()
+
     # ------------------------------------------------------------------
     # Private helpers
     # ------------------------------------------------------------------

src/main.py

@@ -7,7 +7,7 @@ import platform
 import shutil
 import sys
 import traceback
-from datetime import datetime
+from datetime import datetime, timezone
 from pathlib import Path
 
 import click
@@ -62,6 +62,14 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, debug: bool, no_log_file
     """Export ChatGPT and Claude conversations to Markdown for personal archival."""
     ctx.ensure_object(dict)
 
+    # Load .env BEFORE reading any env var (LOG_FILE, CACHE_DIR). Without this
+    # the cache/log paths fell back to defaults regardless of .env — the cache
+    # in particular silently lived at ./cache even when CACHE_DIR was set.
+    # override=False so a real shell env var still wins over .env.
+    import os
+    from dotenv import load_dotenv
+    load_dotenv(override=False)
+
     # Determine console log level
     if debug or verbose:
         level = logging.DEBUG
@@ -71,7 +79,6 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, debug: bool, no_log_file
         level = logging.INFO
 
     # Determine log file path from env (setup_logging handles "none")
-    import os
     log_file = os.getenv("LOG_FILE", "./cache/logs/exporter.log")
 
     setup_logging(level=level, log_file=log_file, no_log_file=no_log_file)
@@ -634,6 +641,16 @@ def _print_doctor_table(checks: list[dict]) -> None:
         "(default: images)."
     ),
 )
+@click.option(
+    "--force",
+    is_flag=True,
+    help=(
+        "Re-export every conversation even if already cached and unchanged. "
+        "Use to re-render the whole archive after a formatting/feature change "
+        "without 'cache --clear' (which would drop Joplin note links). Combine "
+        "with --max-conversations to spread the work over several runs."
+    ),
+)
 @click.option("--dry-run", is_flag=True, help="Show what would be exported without writing anything.")
 @click.pass_context
 def export(
@@ -646,6 +663,7 @@ def export(
     hidden_content: str | None,
     max_conversations: int | None,
     download_media: str | None,
+    force: bool,
     dry_run: bool,
 ) -> None:
     """Export new and updated conversations to Markdown or JSON.
@@ -697,6 +715,18 @@ def export(
     # Session cap: CLI flag wins over MAX_CONVERSATIONS_PER_RUN env default.
     session_cap = max_conversations if max_conversations is not None else cfg.max_conversations
 
+    # Force re-render campaign: stamp a start time so progress can be tracked
+    # across runs. A conversation is "done this campaign" once its exported_at
+    # passes the stamp; finished providers then export nothing, and the
+    # "still awaiting" count shrinks to zero. A fresh campaign starts whenever
+    # none is active (the previous one completed or was never started).
+    campaign_at: str | None = None
+    if force and not dry_run:
+        campaign_at = cache.get_force_campaign()
+        if campaign_at is None:
+            campaign_at = datetime.now(timezone.utc).isoformat()
+            cache.set_force_campaign(campaign_at)
+
     # Determine which providers to run
     providers_to_run = _resolve_providers(provider, cfg)
     if not providers_to_run:
@@ -736,7 +766,9 @@ def export(
                 f"  [dim]--project filter '{project_filter}': {len(all_convs)} matching conversations.[/dim]"
             )
 
-        to_export = cache.get_new_or_updated(prov_name, all_convs)
+        to_export = cache.get_new_or_updated(
+            prov_name, all_convs, force=force, campaign_at=campaign_at
+        )
         skipped = len(all_convs) - len(to_export)
         summary[prov_name]["skipped"] = skipped
 
@@ -758,7 +790,11 @@ def export(
             continue
 
         if not to_export:
-            console.print(f"  [dim]{skipped} conversations already up to date.[/dim]")
+            done_msg = (
+                "already re-rendered this campaign."
+                if force else "conversations already up to date."
+            )
+            console.print(f"  [dim]{skipped} {done_msg}[/dim]")
             continue
 
         if deferred:
@@ -837,14 +873,38 @@ def export(
                     progress.advance(task)
                     continue
 
+        # Force campaign: how many of this provider's conversations still
+        # predate the campaign stamp (i.e. not yet re-rendered this campaign).
+        if force and campaign_at is not None:
+            summary[prov_name]["awaiting"] = sum(
+                1
+                for c in all_convs
+                if cache.exported_at(prov_name, c.get("id") or c.get("uuid", "")) < campaign_at
+            )
+
     if not dry_run:
         _print_export_summary(summary)
-        total_deferred = sum(s.get("deferred", 0) for s in summary.values())
-        if total_deferred:
-            console.print(
-                f"[yellow]{total_deferred} conversation(s) deferred by the session cap "
-                f"({session_cap} per provider per run). Re-run the same command to continue.[/yellow]"
-            )
+        if force:
+            total_awaiting = sum(s.get("awaiting", 0) for s in summary.values())
+            exported_now = sum(s.get("exported", 0) for s in summary.values())
+            if total_awaiting:
+                console.print(
+                    f"[yellow]Force re-render: refreshed {exported_now} this run, "
+                    f"{total_awaiting} still to go. Re-run the same command to continue.[/yellow]"
+                )
+            else:
+                cache.clear_force_campaign()
+                console.print(
+                    "[green]Force re-render complete — every conversation has been "
+                    "re-rendered with the current code.[/green]"
+                )
+        else:
+            total_deferred = sum(s.get("deferred", 0) for s in summary.values())
+            if total_deferred:
+                console.print(
+                    f"[yellow]{total_deferred} conversation(s) deferred by the session cap "
+                    f"({session_cap} per provider per run). Re-run the same command to continue.[/yellow]"
+                )
         # Emit the data-loss summary at INFO level so it lands in the log file
         # AND the operator's console (default level is INFO).
         for line in loss_report.format_summary().split("\n"):

tests/test_cache.py

@@ -151,3 +151,106 @@ class TestGetNewOrUpdated:
         convs = [{"id": "a", "updated_at": "2024-06-01T00:00:00Z"}]
         result = tmp_cache.get_new_or_updated("claude", convs)
         assert len(result) == 1
+
+    def test_force_returns_all_cached_and_unchanged(self, tmp_cache):
+        tmp_cache.mark_exported("claude", "a", {"updated_at": "2024-01-01T00:00:00Z"})
+        convs = [
+            {"id": "a", "updated_at": "2024-01-01T00:00:00Z"},  # cached, unchanged
+            {"id": "b", "updated_at": "2024-01-02T00:00:00Z"},  # new
+        ]
+        assert len(tmp_cache.get_new_or_updated("claude", convs)) == 1
+        assert len(tmp_cache.get_new_or_updated("claude", convs, force=True)) == 2
+
+    def test_force_orders_least_recently_exported_first(self, tmp_cache):
+        # 'a' was exported long ago; 'b' just now; 'c' never.
+        tmp_cache.mark_exported("claude", "a", {"updated_at": "2024-01-01T00:00:00Z"})
+        tmp_cache._data["claude"]["a"]["exported_at"] = "2024-01-01T00:00:00Z"
+        tmp_cache.mark_exported("claude", "b", {"updated_at": "2024-01-01T00:00:00Z"})
+        tmp_cache._data["claude"]["b"]["exported_at"] = "2026-06-12T00:00:00Z"
+        convs = [{"id": "a"}, {"id": "b"}, {"id": "c"}]
+        ordered = [c["id"] for c in tmp_cache.get_new_or_updated("claude", convs, force=True)]
+        # never-exported ('c', exported_at "") and oldest ('a') come before 'b'
+        assert ordered == ["c", "a", "b"]
+
+    def test_force_capped_run_makes_progress(self, tmp_cache):
+        """The bug: force + cap repeated the same head every run. Now each
+        capped run re-exports the next-oldest batch and converges."""
+        convs = [{"id": f"c{i}", "updated_at": "2024-01-01T00:00:00Z"} for i in range(5)]
+        seen = set()
+        for _ in range(3):  # cap=2 over 3 runs should cover all 5
+            batch = tmp_cache.get_new_or_updated("claude", convs, force=True)[:2]
+            for c in batch:
+                seen.add(c["id"])
+                tmp_cache.mark_exported("claude", c["id"], {"file_path": f"/{c['id']}.md"})
+        assert seen == {f"c{i}" for i in range(5)}
+
+    def test_campaign_excludes_already_refreshed(self, tmp_cache):
+        """With a campaign stamp, conversations re-exported during the campaign
+        (exported_at >= stamp) drop out, so the remaining count shrinks to 0."""
+        convs = [{"id": f"c{i}"} for i in range(5)]
+        campaign = "2026-06-12T00:00:00+00:00"
+        # Nothing exported yet → all 5 are candidates.
+        assert len(tmp_cache.get_new_or_updated("claude", convs, force=True,
+                                                campaign_at=campaign)) == 5
+        # Re-export 2 "during" the campaign (after the stamp).
+        for cid in ("c0", "c1"):
+            tmp_cache.mark_exported("claude", cid, {"file_path": f"/{cid}.md"})
+            tmp_cache._data["claude"][cid]["exported_at"] = "2026-06-12T01:00:00+00:00"
+        remaining = tmp_cache.get_new_or_updated("claude", convs, force=True,
+                                                 campaign_at=campaign)
+        assert {c["id"] for c in remaining} == {"c2", "c3", "c4"}
+        # Finish them → none left.
+        for cid in ("c2", "c3", "c4"):
+            tmp_cache.mark_exported("claude", cid, {"file_path": f"/{cid}.md"})
+            tmp_cache._data["claude"][cid]["exported_at"] = "2026-06-12T02:00:00+00:00"
+        assert tmp_cache.get_new_or_updated("claude", convs, force=True,
+                                            campaign_at=campaign) == []
+
+    def test_force_campaign_marker_roundtrip(self, tmp_cache):
+        assert tmp_cache.get_force_campaign() is None
+        tmp_cache.set_force_campaign("2026-06-12T00:00:00+00:00")
+        assert tmp_cache.get_force_campaign() == "2026-06-12T00:00:00+00:00"
+        # Survives reload, and is not mistaken for a provider by stats().
+        reopened = Cache(tmp_cache._dir)
+        assert reopened.get_force_campaign() == "2026-06-12T00:00:00+00:00"
+        assert "force_campaign_at" not in reopened.stats()
+        tmp_cache.clear_force_campaign()
+        assert tmp_cache.get_force_campaign() is None
+
+
+class TestReexportPreservesJoplinLink:
+    """A re-export must not drop the Joplin note link, or re-syncing would
+    create duplicate notes instead of updating the existing ones."""
+
+    def test_joplin_fields_carried_across_reexport(self, tmp_cache):
+        tmp_cache.mark_exported("chatgpt", "c1", {
+            "title": "T", "updated_at": "2024-01-01T00:00:00Z", "file_path": "/x.md",
+        })
+        tmp_cache.mark_joplin_synced("chatgpt", "c1", "note-123")
+        tmp_cache.set_joplin_resources("chatgpt", "c1", {"media/a.png": "res-1"})
+
+        # Re-export the same conversation (new render, new file_path).
+        tmp_cache.mark_exported("chatgpt", "c1", {
+            "title": "T", "updated_at": "2024-01-01T00:00:00Z", "file_path": "/x2.md",
+        })
+
+        entry = tmp_cache.get_all_entries("chatgpt")["c1"]
+        assert entry["joplin_note_id"] == "note-123"
+        assert entry["joplin_resources"] == {"media/a.png": "res-1"}
+        assert entry["file_path"] == "/x2.md"
+
+    def test_reexport_marks_note_for_update_not_create(self, tmp_cache):
+        tmp_cache.mark_exported("chatgpt", "c1", {
+            "updated_at": "2024-01-01T00:00:00Z", "file_path": "/x.md",
+        })
+        tmp_cache.mark_joplin_synced("chatgpt", "c1", "note-123")
+        # Nothing pending right after sync.
+        assert tmp_cache.get_joplin_pending("chatgpt") == []
+        # Re-export bumps exported_at past joplin_synced_at → pending for update,
+        # carrying the existing note id so the sync updates rather than creates.
+        tmp_cache.mark_exported("chatgpt", "c1", {
+            "updated_at": "2024-01-01T00:00:00Z", "file_path": "/x.md",
+        })
+        pending = tmp_cache.get_joplin_pending("chatgpt")
+        assert len(pending) == 1
+        assert pending[0][1]["joplin_note_id"] == "note-123"