Compare commits
5
Commits
304cf4fde4
...
v0.2.1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
19bfdaecbe | ||
|
|
4ccd918eb1 | ||
|
|
a869e8c7ba | ||
|
|
340293ab94 | ||
|
|
050cd49124 |
+8
-5
@@ -6,9 +6,12 @@
|
||||
|
||||
# --- ChatGPT ---
|
||||
# How to get: open chatgpt.com in Chrome → F12 → Application tab
|
||||
# → Cookies → https://chatgpt.com → find "__Secure-next-auth.session-token" → copy Value
|
||||
# Token type: JWT (starts with "eyJ"). Typically valid for ~7 days.
|
||||
# → Cookies → https://chatgpt.com → find the two cookie chunks:
|
||||
# __Secure-next-auth.session-token.0 (starts with "eyJ") → CHATGPT_SESSION_TOKEN
|
||||
# __Secure-next-auth.session-token.1 (the remainder) → CHATGPT_SESSION_TOKEN_1
|
||||
# Token type: JWE. Typically valid for ~7 days.
|
||||
CHATGPT_SESSION_TOKEN=
|
||||
CHATGPT_SESSION_TOKEN_1=
|
||||
|
||||
# ChatGPT Projects (optional): comma-separated list of project gizmo IDs.
|
||||
# Project conversations are NOT included in the default /conversations listing.
|
||||
@@ -46,9 +49,9 @@ JOPLIN_API_URL=http://localhost:41184
|
||||
# JOPLIN_REQUEST_TIMEOUT=30
|
||||
|
||||
# --- Cache ---
|
||||
# Where the sync manifest and logs are stored (default: ~/.ai-chat-exporter)
|
||||
CACHE_DIR=~/.ai-chat-exporter
|
||||
# Where the sync manifest is stored (default: ./cache, inside the install directory)
|
||||
CACHE_DIR=./cache
|
||||
|
||||
# --- Logging ---
|
||||
# Log file path. Set to "none" to disable file logging.
|
||||
LOG_FILE=~/.ai-chat-exporter/logs/exporter.log
|
||||
LOG_FILE=./cache/logs/exporter.log
|
||||
|
||||
@@ -25,10 +25,14 @@ exports/
|
||||
!CHANGELOG.md
|
||||
|
||||
# Cache and logs
|
||||
cache/
|
||||
.ai-chat-exporter/
|
||||
logs/
|
||||
*.log
|
||||
|
||||
# Test tracking
|
||||
test-plan.csv
|
||||
|
||||
# Editor / OS
|
||||
.DS_Store
|
||||
.idea/
|
||||
|
||||
@@ -28,6 +28,8 @@ This tool is designed for a single user backing up their own conversations. Do n
|
||||
|
||||
## Installation
|
||||
|
||||
### Linux / macOS
|
||||
|
||||
```bash
|
||||
git clone <repo-url>
|
||||
cd ai-chat-exporter
|
||||
@@ -36,6 +38,37 @@ source .venv/bin/activate
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
### Windows
|
||||
|
||||
No admin access required. Run these in **Command Prompt** (`cmd.exe`) — it's the simplest option on Windows because it doesn't have PowerShell's script execution policy restrictions.
|
||||
|
||||
```bat
|
||||
git clone <repo-url>
|
||||
cd ai-chat-exporter
|
||||
python -m venv .venv
|
||||
.venv\Scripts\activate
|
||||
pip install -e ".[dev]"
|
||||
```
|
||||
|
||||
All `ai-chat-exporter` commands work identically in Command Prompt.
|
||||
|
||||
**Using PowerShell instead?** If you prefer PowerShell, you may need to allow script execution first (one-time, current user only):
|
||||
|
||||
```powershell
|
||||
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
|
||||
```
|
||||
|
||||
Then activate the venv and run commands the same way.
|
||||
|
||||
**Prerequisites:**
|
||||
- Python 3.11 or later — install from [python.org](https://www.python.org/downloads/windows/). During installation, tick **"Add Python to PATH"**.
|
||||
- Git — install from [git-scm.com](https://git-scm.com/) if not already present.
|
||||
|
||||
**Notes:**
|
||||
- The cache manifest and logs are stored in `cache\` inside the install directory — the same as on Linux.
|
||||
- File permission hardening (`chmod 600`) is silently ignored on Windows — not a concern for single-user desktop use.
|
||||
- Joplin Web Clipper runs on `localhost:41184` on all platforms; no configuration changes needed.
|
||||
|
||||
---
|
||||
|
||||
## First Run: Run Doctor
|
||||
@@ -43,7 +76,7 @@ pip install -e ".[dev]"
|
||||
Before anything else, validate your setup:
|
||||
|
||||
```bash
|
||||
python -m src.main doctor
|
||||
ai-chat-exporter doctor
|
||||
```
|
||||
|
||||
This checks token presence, format, expiry, directory permissions, disk space, and live API connectivity. Fix any failures before proceeding.
|
||||
@@ -76,7 +109,7 @@ Session tokens are how your browser stays logged in. This tool uses them to acce
|
||||
### When Tokens Expire
|
||||
|
||||
When a token expires you'll see a `401 Unauthorized` error. To refresh:
|
||||
- Re-run the `auth` wizard: `python -m src.main auth`
|
||||
- Re-run the `auth` wizard: `ai-chat-exporter auth`
|
||||
- Or manually update the value in your `.env` file
|
||||
|
||||
---
|
||||
@@ -86,7 +119,7 @@ When a token expires you'll see a `401 Unauthorized` error. To refresh:
|
||||
The easiest way to configure tokens is the interactive wizard:
|
||||
|
||||
```bash
|
||||
python -m src.main auth
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
This walks you through finding your token, validates it, shows the expiry date (ChatGPT only), and offers to write it to your `.env` automatically. Tokens are never echoed to the terminal.
|
||||
@@ -128,8 +161,8 @@ cp .env.example .env
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `CACHE_DIR` | `~/.ai-chat-exporter` | Where to store the sync manifest |
|
||||
| `LOG_FILE` | `~/.ai-chat-exporter/logs/exporter.log` | Log file path (`none` to disable) |
|
||||
| `CACHE_DIR` | `./cache` | Where to store the sync manifest |
|
||||
| `LOG_FILE` | `./cache/logs/exporter.log` | Log file path (`none` to disable) |
|
||||
|
||||
---
|
||||
|
||||
@@ -218,7 +251,7 @@ Each provider+project combination maps to a flat Joplin notebook created automat
|
||||
### `auth` — Interactive token setup
|
||||
|
||||
```bash
|
||||
python -m src.main auth
|
||||
ai-chat-exporter auth
|
||||
```
|
||||
|
||||
Guided wizard to find and save session tokens and ChatGPT project IDs. Detects OS and shows the correct DevTools shortcut.
|
||||
@@ -226,7 +259,7 @@ Guided wizard to find and save session tokens and ChatGPT project IDs. Detects O
|
||||
### `doctor` — Health check
|
||||
|
||||
```bash
|
||||
python -m src.main doctor
|
||||
ai-chat-exporter doctor
|
||||
```
|
||||
|
||||
Checks: token presence, JWT validity and expiry, directory permissions, disk space, live API reachability. Exits with code 0 if all pass, 1 if any fail.
|
||||
@@ -235,31 +268,31 @@ Checks: token presence, JWT validity and expiry, directory permissions, disk spa
|
||||
|
||||
```bash
|
||||
# Export everything (new/updated only)
|
||||
python -m src.main export
|
||||
ai-chat-exporter export
|
||||
|
||||
# Single provider
|
||||
python -m src.main export --provider claude
|
||||
ai-chat-exporter export --provider claude
|
||||
|
||||
# JSON output
|
||||
python -m src.main export --format json
|
||||
ai-chat-exporter export --format json
|
||||
|
||||
# Both Markdown and JSON
|
||||
python -m src.main export --format both
|
||||
ai-chat-exporter export --format both
|
||||
|
||||
# Only conversations updated since a date
|
||||
python -m src.main export --since 2024-06-01
|
||||
ai-chat-exporter export --since 2024-06-01
|
||||
|
||||
# Only conversations in a specific project (case-insensitive substring)
|
||||
python -m src.main export --project "learning python"
|
||||
ai-chat-exporter export --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
python -m src.main export --project none
|
||||
ai-chat-exporter export --project none
|
||||
|
||||
# Write to a custom directory
|
||||
python -m src.main export --output /path/to/my/notes
|
||||
ai-chat-exporter export --output /path/to/my/notes
|
||||
|
||||
# Preview without writing anything
|
||||
python -m src.main export --dry-run
|
||||
ai-chat-exporter export --dry-run
|
||||
```
|
||||
|
||||
Options: `--provider [chatgpt|claude|all]`, `--format [markdown|json|both]`, `--output PATH`, `--since YYYY-MM-DD`, `--project NAME`, `--dry-run`
|
||||
@@ -268,16 +301,16 @@ Options: `--provider [chatgpt|claude|all]`, `--format [markdown|json|both]`, `--
|
||||
|
||||
```bash
|
||||
# List all conversations for all providers
|
||||
python -m src.main list
|
||||
ai-chat-exporter list
|
||||
|
||||
# Single provider
|
||||
python -m src.main list --provider chatgpt
|
||||
ai-chat-exporter list --provider chatgpt
|
||||
|
||||
# Filter by project
|
||||
python -m src.main list --project "learning python"
|
||||
ai-chat-exporter list --project "learning python"
|
||||
|
||||
# Only conversations outside any project
|
||||
python -m src.main list --project none
|
||||
ai-chat-exporter list --project none
|
||||
```
|
||||
|
||||
Fetches and displays all conversations without exporting them. Useful for verifying what the tool can see before running an export.
|
||||
@@ -286,19 +319,19 @@ Fetches and displays all conversations without exporting them. Useful for verify
|
||||
|
||||
```bash
|
||||
# Sync all pending conversations to Joplin
|
||||
python -m src.main joplin
|
||||
ai-chat-exporter joplin
|
||||
|
||||
# Preview what would be synced without sending anything
|
||||
python -m src.main joplin --dry-run
|
||||
ai-chat-exporter joplin --dry-run
|
||||
|
||||
# Sync a single provider
|
||||
python -m src.main joplin --provider chatgpt
|
||||
ai-chat-exporter joplin --provider chatgpt
|
||||
|
||||
# Sync only conversations in a specific project
|
||||
python -m src.main joplin --project "learning python"
|
||||
ai-chat-exporter joplin --project "learning python"
|
||||
|
||||
# Sync only conversations outside any project
|
||||
python -m src.main joplin --project none
|
||||
ai-chat-exporter joplin --project none
|
||||
```
|
||||
|
||||
Reads the local export cache and pushes each exported Markdown file to Joplin as a note. Notebooks are created automatically. Re-running is safe — notes are updated (not duplicated).
|
||||
@@ -315,20 +348,20 @@ Options: `--provider [chatgpt|claude|all]`, `--project NAME`, `--dry-run`
|
||||
|
||||
```bash
|
||||
# Show statistics
|
||||
python -m src.main cache --show
|
||||
ai-chat-exporter cache --show
|
||||
|
||||
# Clear all cached entries (forces full re-export next run)
|
||||
python -m src.main cache --clear
|
||||
ai-chat-exporter cache --clear
|
||||
|
||||
# Clear a single provider
|
||||
python -m src.main cache --clear --provider claude
|
||||
ai-chat-exporter cache --clear --provider claude
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## How the Cache Works
|
||||
|
||||
The cache manifest lives at `~/.ai-chat-exporter/manifest.json` and records every exported conversation: its title, project, `updated_at` timestamp, output file path, and (after Joplin sync) the Joplin note ID.
|
||||
The cache manifest lives at `cache/manifest.json` (inside the install directory) and records every exported conversation: its title, project, `updated_at` timestamp, output file path, and (after Joplin sync) the Joplin note ID.
|
||||
|
||||
On every `export` run:
|
||||
1. Fetch the full conversation list from the provider
|
||||
@@ -343,7 +376,7 @@ On every `joplin` run:
|
||||
|
||||
**This design makes every run inherently resumable.** If the tool is interrupted for any reason — rate limit, network drop, Ctrl+C, crash — simply re-run the same command. It will skip already-processed conversations and continue from where it stopped.
|
||||
|
||||
To force a full re-export: `python -m src.main cache --clear` then re-run export.
|
||||
To force a full re-export: `ai-chat-exporter cache --clear` then re-run export.
|
||||
|
||||
---
|
||||
|
||||
@@ -351,7 +384,7 @@ To force a full re-export: `python -m src.main cache --clear` then re-run export
|
||||
|
||||
### `401 Unauthorized`
|
||||
Your session token has expired.
|
||||
- Run `python -m src.main auth` to get a new token interactively
|
||||
- Run `ai-chat-exporter auth` to get a new token interactively
|
||||
- Or manually copy a fresh cookie value into your `.env` file
|
||||
|
||||
Note: Claude's `sessionKey` is an opaque string — the only way to know it's expired is the 401 error. ChatGPT JWTs have an `exp` claim that the `doctor` command can decode and display.
|
||||
@@ -391,10 +424,10 @@ The provider's internal API may have changed. Run with `--debug`, sanitize the o
|
||||
Images, code interpreter outputs, DALL-E generations, and Claude artifacts are not exported in v0.2.0. A WARNING is logged for each skipped item. See `FUTURE.md` for the roadmap.
|
||||
|
||||
### Empty export / all conversations skipped
|
||||
No new or updated conversations since your last run. To verify: `python -m src.main cache --show`. To force a full re-export: `python -m src.main cache --clear`.
|
||||
No new or updated conversations since your last run. To verify: `ai-chat-exporter cache --show`. To force a full re-export: `ai-chat-exporter cache --clear`.
|
||||
|
||||
### Filing a bug report
|
||||
1. Run with `--debug`: `python -m src.main export --debug 2>&1 | tee debug.log`
|
||||
1. Run with `--debug`: `ai-chat-exporter export --debug 2>&1 | tee debug.log`
|
||||
2. Remove any personal conversation content from `debug.log`
|
||||
3. Open a GitHub Issue with the sanitized log and the exact command you ran
|
||||
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "ai-chat-exporter"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
description = "Export ChatGPT and Claude conversation history to Markdown for personal archival in Joplin"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
|
||||
+7
-4
@@ -28,6 +28,7 @@ class ConfigError(Exception):
|
||||
@dataclass
|
||||
class Config:
|
||||
chatgpt_session_token: str | None
|
||||
chatgpt_session_token_1: str | None
|
||||
claude_session_key: str | None
|
||||
export_dir: Path
|
||||
output_structure: str
|
||||
@@ -55,11 +56,12 @@ def load_config() -> Config:
|
||||
load_dotenv(override=False)
|
||||
|
||||
chatgpt_token = os.getenv("CHATGPT_SESSION_TOKEN", "").strip() or None
|
||||
chatgpt_token_1 = os.getenv("CHATGPT_SESSION_TOKEN_1", "").strip() or None
|
||||
claude_key = os.getenv("CLAUDE_SESSION_KEY", "").strip() or None
|
||||
export_dir = Path(os.getenv("EXPORT_DIR", "./exports")).expanduser()
|
||||
output_structure = os.getenv("OUTPUT_STRUCTURE", "provider/project/year").strip()
|
||||
cache_dir = Path(os.getenv("CACHE_DIR", "~/.ai-chat-exporter")).expanduser()
|
||||
log_file = os.getenv("LOG_FILE", "~/.ai-chat-exporter/logs/exporter.log").strip()
|
||||
cache_dir = Path(os.getenv("CACHE_DIR", "./cache")).expanduser()
|
||||
log_file = os.getenv("LOG_FILE", "./cache/logs/exporter.log").strip()
|
||||
|
||||
# Joplin
|
||||
joplin_token = os.getenv("JOPLIN_API_TOKEN", "").strip() or None
|
||||
@@ -101,7 +103,7 @@ def load_config() -> Config:
|
||||
if not chatgpt_token and not claude_key:
|
||||
logger.warning(
|
||||
"Neither CHATGPT_SESSION_TOKEN nor CLAUDE_SESSION_KEY is set. "
|
||||
"Run 'python -m src.main auth' to configure credentials."
|
||||
"Run 'ai-chat-exporter auth' to configure credentials."
|
||||
)
|
||||
|
||||
# Create and validate output directory
|
||||
@@ -127,6 +129,7 @@ def load_config() -> Config:
|
||||
|
||||
config = Config(
|
||||
chatgpt_session_token=chatgpt_token,
|
||||
chatgpt_session_token_1=chatgpt_token_1,
|
||||
claude_session_key=claude_key,
|
||||
export_dir=export_dir,
|
||||
output_structure=output_structure,
|
||||
@@ -173,7 +176,7 @@ def _validate_chatgpt_token(token: str) -> datetime | None:
|
||||
if delta.total_seconds() < 0:
|
||||
logger.warning(
|
||||
"CHATGPT_SESSION_TOKEN expired at %s. "
|
||||
"Run 'python -m src.main auth' to refresh it.",
|
||||
"Run 'ai-chat-exporter auth' to refresh it.",
|
||||
expiry.strftime("%Y-%m-%d %H:%M UTC"),
|
||||
)
|
||||
elif delta.total_seconds() < 86400:
|
||||
|
||||
+81
-18
@@ -70,7 +70,7 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, debug: bool, no_log_file
|
||||
|
||||
# Determine log file path from env (setup_logging handles "none")
|
||||
import os
|
||||
log_file = os.getenv("LOG_FILE", "~/.ai-chat-exporter/logs/exporter.log")
|
||||
log_file = os.getenv("LOG_FILE", "./cache/logs/exporter.log")
|
||||
|
||||
setup_logging(level=level, log_file=log_file, no_log_file=no_log_file)
|
||||
|
||||
@@ -79,7 +79,7 @@ def cli(ctx: click.Context, verbose: bool, quiet: bool, debug: bool, no_log_file
|
||||
|
||||
# Initialise cache (needed for ToS gate on every command)
|
||||
import os
|
||||
cache_dir = Path(os.getenv("CACHE_DIR", "~/.ai-chat-exporter")).expanduser()
|
||||
cache_dir = Path(os.getenv("CACHE_DIR", "./cache")).expanduser()
|
||||
try:
|
||||
cache = Cache(cache_dir)
|
||||
except CacheError as e:
|
||||
@@ -140,7 +140,7 @@ def auth(ctx: click.Context) -> None:
|
||||
if configure_claude:
|
||||
_auth_claude(os_name)
|
||||
|
||||
console.print("\n[green]Done! Run 'python -m src.main doctor' to verify your setup.[/green]")
|
||||
console.print("\n[green]Done! Run 'ai-chat-exporter doctor' to verify your setup.[/green]")
|
||||
|
||||
|
||||
def _auth_chatgpt(os_name: str) -> None:
|
||||
@@ -153,15 +153,19 @@ def _auth_chatgpt(os_name: str) -> None:
|
||||
else:
|
||||
console.print("2. Press [bold]F12[/bold] to open DevTools → Application tab.")
|
||||
console.print("3. Expand [bold]Cookies[/bold] → [bold]https://chatgpt.com[/bold]")
|
||||
console.print("4. Find [bold]__Secure-next-auth.session-token[/bold] → copy the Value.")
|
||||
console.print(" (Token starts with 'eyJ...' — it is a long JWT string)")
|
||||
console.print("5. Paste it below (input is hidden).\n")
|
||||
console.print("4. ChatGPT splits the session token across two cookies:")
|
||||
console.print(" [bold]__Secure-next-auth.session-token.0[/bold] (starts with 'eyJ')")
|
||||
console.print(" [bold]__Secure-next-auth.session-token.1[/bold] (the remainder)")
|
||||
console.print(" Copy each Value in turn and paste below.")
|
||||
console.print(" (If you only see one cookie without a .0/.1 suffix, paste it for .0 and leave .1 blank.)\n")
|
||||
|
||||
token = click.prompt("ChatGPT session token", hide_input=True, default="", show_default=False).strip()
|
||||
token = click.prompt("ChatGPT session token (.0)", hide_input=True, default="", show_default=False).strip()
|
||||
if not token:
|
||||
console.print("[yellow]Skipped ChatGPT token.[/yellow]")
|
||||
return
|
||||
|
||||
token_1 = click.prompt("ChatGPT session token (.1, leave blank if absent)", hide_input=True, default="", show_default=False).strip() or None
|
||||
|
||||
# Validate
|
||||
if not token.startswith("eyJ"):
|
||||
console.print("[yellow]Warning: token doesn't look like a JWT (expected 'eyJ...').[/yellow]")
|
||||
@@ -178,7 +182,28 @@ def _auth_chatgpt(os_name: str) -> None:
|
||||
except Exception:
|
||||
console.print("[yellow]Could not decode token expiry.[/yellow]")
|
||||
|
||||
# Live validation — exchange session token for an access token
|
||||
_valid = False
|
||||
_error: str | None = None
|
||||
with console.status("[dim]Validating token with ChatGPT API…[/dim]"):
|
||||
try:
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
_prov = ChatGPTProvider(session_token=token, session_token_1=token_1)
|
||||
_prov._fetch_access_token()
|
||||
_valid = True
|
||||
except ProviderError as e:
|
||||
_error = str(e.original)
|
||||
except Exception as e:
|
||||
_error = str(e)
|
||||
|
||||
if _valid:
|
||||
console.print("[green]✓ Token verified — connected to ChatGPT API.[/green]")
|
||||
else:
|
||||
console.print(f"[red]✗ Token validation failed: {_error}[/red]")
|
||||
|
||||
_write_token_to_env("CHATGPT_SESSION_TOKEN", token)
|
||||
if token_1:
|
||||
_write_token_to_env("CHATGPT_SESSION_TOKEN_1", token_1)
|
||||
|
||||
# --- ChatGPT Projects ---
|
||||
console.print("\n[bold]ChatGPT Projects (optional)[/bold]")
|
||||
@@ -231,7 +256,25 @@ def _auth_claude(os_name: str) -> None:
|
||||
console.print("[yellow]Skipped Claude token.[/yellow]")
|
||||
return
|
||||
|
||||
console.print("[green]Claude session key saved.[/green]")
|
||||
# Live validation — fetch org ID (the first call any Claude operation makes)
|
||||
_valid = False
|
||||
_error: str | None = None
|
||||
with console.status("[dim]Validating token with Claude API…[/dim]"):
|
||||
try:
|
||||
from src.providers.claude import ClaudeProvider
|
||||
_prov = ClaudeProvider(session_key=key)
|
||||
_prov._get_org_id()
|
||||
_valid = True
|
||||
except ProviderError as e:
|
||||
_error = str(e.original)
|
||||
except Exception as e:
|
||||
_error = str(e)
|
||||
|
||||
if _valid:
|
||||
console.print("[green]✓ Token verified — connected to Claude API.[/green]")
|
||||
else:
|
||||
console.print(f"[red]✗ Token validation failed: {_error}[/red]")
|
||||
|
||||
_write_token_to_env("CLAUDE_SESSION_KEY", key)
|
||||
|
||||
|
||||
@@ -341,7 +384,7 @@ def _run_doctor_checks() -> list[dict]:
|
||||
|
||||
# Directories
|
||||
export_dir = Path(os.getenv("EXPORT_DIR", "./exports")).expanduser()
|
||||
cache_dir = Path(os.getenv("CACHE_DIR", "~/.ai-chat-exporter")).expanduser()
|
||||
cache_dir = Path(os.getenv("CACHE_DIR", "./cache")).expanduser()
|
||||
|
||||
for label, dirpath in [("Export dir writable", export_dir), ("Cache dir writable", cache_dir)]:
|
||||
try:
|
||||
@@ -365,7 +408,8 @@ def _run_doctor_checks() -> list[dict]:
|
||||
if chatgpt_token:
|
||||
try:
|
||||
from src.providers.chatgpt import ChatGPTProvider
|
||||
p = ChatGPTProvider(chatgpt_token)
|
||||
chatgpt_token_1 = os.getenv("CHATGPT_SESSION_TOKEN_1", "").strip() or None
|
||||
p = ChatGPTProvider(chatgpt_token, session_token_1=chatgpt_token_1)
|
||||
results = p.list_conversations(offset=0, limit=1)
|
||||
add("ChatGPT API reachable", True, f"Got {len(results)} result(s)")
|
||||
except ProviderError as e:
|
||||
@@ -496,7 +540,7 @@ def export(
|
||||
providers_to_run = _resolve_providers(provider, cfg)
|
||||
if not providers_to_run:
|
||||
err_console.print(
|
||||
"[red]No providers configured. Run 'python -m src.main auth' to set up tokens.[/red]"
|
||||
"[red]No providers configured. Run 'ai-chat-exporter auth' to set up tokens.[/red]"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
@@ -557,6 +601,16 @@ def export(
|
||||
conv_id = raw_conv.get("id") or raw_conv.get("uuid", "unknown")
|
||||
try:
|
||||
full_raw = prov_instance.get_conversation(conv_id)
|
||||
# Propagate metadata from the listing summary into the full
|
||||
# detail so normalize_conversation can use it.
|
||||
# - Keys starting with "_" are provider annotations
|
||||
# (e.g. _project_name injected by ChatGPT project fetching).
|
||||
# - "project" is included explicitly because Claude's detail
|
||||
# endpoint omits it even though the listing returns it.
|
||||
_PROPAGATE_KEYS = {"project"}
|
||||
for key, val in raw_conv.items():
|
||||
if (key.startswith("_") or key in _PROPAGATE_KEYS) and key not in full_raw:
|
||||
full_raw[key] = val
|
||||
normalized = prov_instance.normalize_conversation(full_raw)
|
||||
|
||||
exported_path: Path | None = None
|
||||
@@ -618,6 +672,7 @@ def _resolve_providers(provider: str, cfg) -> list[tuple[str, object]]:
|
||||
"chatgpt",
|
||||
ChatGPTProvider(
|
||||
session_token=cfg.chatgpt_session_token,
|
||||
session_token_1=cfg.chatgpt_session_token_1,
|
||||
project_ids=cfg.chatgpt_project_ids,
|
||||
),
|
||||
))
|
||||
@@ -757,18 +812,26 @@ def list_conversations(ctx: click.Context, provider: str, project_filter: str |
|
||||
if project_filter is not None:
|
||||
all_convs = _filter_by_project(all_convs, project_filter)
|
||||
|
||||
table = Table()
|
||||
table.add_column("Title")
|
||||
table.add_column("Project")
|
||||
table.add_column("Updated")
|
||||
table.add_column("ID")
|
||||
# no_wrap + overflow="ellipsis" prevents Rich from wrapping cells to
|
||||
# multiple lines on narrow terminals (e.g. Windows Command Prompt),
|
||||
# which can otherwise make the output look garbled. Widths are tuned
|
||||
# to fit within an 80-column terminal.
|
||||
# Total width budget for 80-column terminals:
|
||||
# borders (5) + padding (4 cols * 2) = 13 chars of overhead
|
||||
# remaining 67 chars split: 34 title + 15 project + 10 date + 8 id
|
||||
table = Table(show_lines=False, expand=False, padding=(0, 1))
|
||||
table.add_column("Title", no_wrap=True, overflow="ellipsis", max_width=34)
|
||||
table.add_column("Project", no_wrap=True, overflow="ellipsis", max_width=15)
|
||||
table.add_column("Updated", no_wrap=True, min_width=10)
|
||||
table.add_column("ID", no_wrap=True, min_width=8)
|
||||
|
||||
for conv in all_convs:
|
||||
title = conv.get("title") or "Untitled"
|
||||
# ChatGPT uses "title"; Claude uses "name".
|
||||
title = conv.get("title") or conv.get("name") or "Untitled"
|
||||
project = _raw_project_name(conv) or ""
|
||||
updated = (conv.get("updated_at") or conv.get("update_time") or "")[:10]
|
||||
conv_id = (conv.get("id") or conv.get("uuid") or "")[:8]
|
||||
table.add_row(title[:60], project[:30], updated, conv_id)
|
||||
table.add_row(title, project, updated, conv_id)
|
||||
|
||||
console.print(table)
|
||||
console.print(f"Total: {len(all_convs)} conversations")
|
||||
|
||||
@@ -326,7 +326,7 @@ class BaseProvider(ABC):
|
||||
msg = (
|
||||
f"[{self.provider_name}] Authentication failed (401 Unauthorized). "
|
||||
"Your session token has likely expired. "
|
||||
"Run 'python -m src.main auth' to refresh your token."
|
||||
"Run 'ai-chat-exporter auth' to refresh your token."
|
||||
)
|
||||
logger.error(msg)
|
||||
raise ProviderError(
|
||||
|
||||
+85
-27
@@ -56,6 +56,7 @@ class ChatGPTProvider(BaseProvider):
|
||||
def __init__(
|
||||
self,
|
||||
session_token: str | None = None,
|
||||
session_token_1: str | None = None,
|
||||
project_ids: list[str] | None = None,
|
||||
) -> None:
|
||||
# Pass a curl_cffi session to the base class instead of a requests.Session.
|
||||
@@ -77,11 +78,15 @@ class ChatGPTProvider(BaseProvider):
|
||||
"init",
|
||||
RuntimeError(
|
||||
"CHATGPT_SESSION_TOKEN is not set. "
|
||||
"Run 'python -m src.main auth' to configure it."
|
||||
"Run 'ai-chat-exporter auth' to configure it."
|
||||
),
|
||||
)
|
||||
self._session_token = token
|
||||
|
||||
# Second chunk of the session token (ChatGPT splits large cookies into
|
||||
# __Secure-next-auth.session-token.0 and .1 to stay under the 4KB limit).
|
||||
token_1 = session_token_1 or os.getenv("CHATGPT_SESSION_TOKEN_1", "").strip() or None
|
||||
|
||||
# Project gizmo IDs (g-p-xxx) whose conversations we'll fetch.
|
||||
# ChatGPT project conversations do not appear in the default
|
||||
# /conversations listing — they require explicit project IDs.
|
||||
@@ -93,13 +98,24 @@ class ChatGPTProvider(BaseProvider):
|
||||
# Cache of project_id → display name (avoids re-fetching gizmo details)
|
||||
self._project_name_cache: dict[str, str] = {}
|
||||
|
||||
# Set the session cookie in the cookie jar
|
||||
# ChatGPT now splits large session cookies into .0 / .1 chunks.
|
||||
# Always send both named chunks; the server reassembles them.
|
||||
self._session.cookies.set(
|
||||
"__Secure-next-auth.session-token",
|
||||
"__Secure-next-auth.session-token.0",
|
||||
token,
|
||||
domain="chatgpt.com",
|
||||
path="/",
|
||||
)
|
||||
if token_1:
|
||||
self._session.cookies.set(
|
||||
"__Secure-next-auth.session-token.1",
|
||||
token_1,
|
||||
domain="chatgpt.com",
|
||||
path="/",
|
||||
)
|
||||
logger.debug("[chatgpt] Set both session cookie chunks (.0 and .1)")
|
||||
else:
|
||||
logger.debug("[chatgpt] Set session cookie chunk .0 only (no .1 configured)")
|
||||
|
||||
# Set only Referer and sec-fetch-* headers for the auth exchange.
|
||||
# Origin is intentionally omitted: Chrome does not send Origin on
|
||||
@@ -157,7 +173,7 @@ class ChatGPTProvider(BaseProvider):
|
||||
"fetch_access_token",
|
||||
RuntimeError(
|
||||
"No accessToken in /api/auth/session response. "
|
||||
"Your session token may be expired — run 'python -m src.main auth' to refresh."
|
||||
"Your session token may be expired — run 'ai-chat-exporter auth' to refresh."
|
||||
),
|
||||
)
|
||||
return access_token
|
||||
@@ -169,7 +185,7 @@ class ChatGPTProvider(BaseProvider):
|
||||
"The session token is used to obtain a short-lived access token via /api/auth/session. "
|
||||
"To refresh: open chatgpt.com in Chrome → F12 → Application → Cookies "
|
||||
"→ find '__Secure-next-auth.session-token' → copy the value. "
|
||||
"Then run 'python -m src.main auth' or update CHATGPT_SESSION_TOKEN in .env."
|
||||
"Then run 'ai-chat-exporter auth' or update CHATGPT_SESSION_TOKEN in .env."
|
||||
)
|
||||
logger.error(msg)
|
||||
raise ProviderError(
|
||||
@@ -369,7 +385,7 @@ class ChatGPTProvider(BaseProvider):
|
||||
logger.info(
|
||||
"[chatgpt] No project IDs configured — skipping project conversations. "
|
||||
"To include projects, set CHATGPT_PROJECT_IDS in .env "
|
||||
"(see 'python -m src.main auth' for instructions)."
|
||||
"(see 'ai-chat-exporter auth' for instructions)."
|
||||
)
|
||||
return self._apply_since_filter(default_convs, since)
|
||||
|
||||
@@ -551,12 +567,16 @@ class ChatGPTProvider(BaseProvider):
|
||||
created_at = _ts_to_iso(raw.get("create_time"))
|
||||
updated_at = _ts_to_iso(raw.get("update_time"))
|
||||
|
||||
# Look up project name from the map built during fetch_all_conversations.
|
||||
project = self._project_map.get(conv_id) if conv_id else None
|
||||
# Prefer _project_name annotation injected from the listing summary
|
||||
# (propagated by the export loop). Fall back to _project_map lookup.
|
||||
project = raw.get("_project_name") or (
|
||||
self._project_map.get(conv_id) if conv_id else None
|
||||
)
|
||||
logger.debug(
|
||||
"[chatgpt] normalize_conversation[%s]: project_map lookup → %r",
|
||||
"[chatgpt] normalize_conversation[%s]: project=%r (source=%s)",
|
||||
conv_id[:8] if conv_id else "?",
|
||||
project,
|
||||
"_project_name" if raw.get("_project_name") else "_project_map",
|
||||
)
|
||||
|
||||
mapping: dict = raw.get("mapping", {})
|
||||
@@ -622,18 +642,22 @@ def _extract_messages(
|
||||
if role in ("user", "assistant"):
|
||||
content_obj = msg_data.get("content", {})
|
||||
content_type = content_obj.get("content_type", "text")
|
||||
text = _extract_text(content_obj, conv_id, node_id)
|
||||
|
||||
if content_type != "text":
|
||||
logger.warning(
|
||||
"[chatgpt] Skipping %s content in conversation %s message %s "
|
||||
"— rich content not yet supported (see FUTURE.md)",
|
||||
content_type,
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
elif text:
|
||||
ts = msg_data.get("create_time")
|
||||
|
||||
# Content types whose parts[] contain plain text strings.
|
||||
# model_editable_context / user_editable_context = project instructions
|
||||
# thoughts / reasoning_recap = o1/o3 reasoning traces
|
||||
_TEXT_PARTS_TYPES = {
|
||||
"text",
|
||||
"model_editable_context",
|
||||
"user_editable_context",
|
||||
"thoughts",
|
||||
"reasoning_recap",
|
||||
}
|
||||
|
||||
if content_type in _TEXT_PARTS_TYPES:
|
||||
text = _extract_text(content_obj, conv_id, node_id)
|
||||
if text:
|
||||
messages.append(
|
||||
{
|
||||
"role": role,
|
||||
@@ -644,7 +668,32 @@ def _extract_messages(
|
||||
)
|
||||
else:
|
||||
logger.debug(
|
||||
"[chatgpt] Skipping empty message in conversation %s", conv_id[:8]
|
||||
"[chatgpt] Skipping empty %s message in conversation %s",
|
||||
content_type,
|
||||
conv_id[:8],
|
||||
)
|
||||
elif content_type == "code":
|
||||
# Inline code response — extract and wrap in a fenced code block
|
||||
code_text = content_obj.get("text") or "\n".join(
|
||||
p for p in content_obj.get("parts", []) if isinstance(p, str)
|
||||
)
|
||||
language = content_obj.get("language", "")
|
||||
if code_text:
|
||||
messages.append(
|
||||
{
|
||||
"role": role,
|
||||
"content": f"```{language}\n{code_text}\n```",
|
||||
"content_type": "code",
|
||||
"timestamp": _ts_to_iso(ts) if ts else None,
|
||||
}
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"[chatgpt] Skipping %s content in conversation %s message %s "
|
||||
"— rich content not yet supported (see FUTURE.md)",
|
||||
content_type,
|
||||
conv_id[:8],
|
||||
node_id[:8],
|
||||
)
|
||||
|
||||
# Walk children in order (ChatGPT typically has one child per node in a linear chat)
|
||||
@@ -670,7 +719,13 @@ def _find_root(mapping: dict[str, Any]) -> str | None:
|
||||
|
||||
|
||||
def _extract_text(content_obj: dict, conv_id: str, node_id: str) -> str:
|
||||
"""Extract plain text from a ChatGPT content object."""
|
||||
"""Extract plain text from a ChatGPT content object.
|
||||
|
||||
Handles three part shapes:
|
||||
- str — plain text (most messages)
|
||||
- dict with content_type="text" — wrapped text part
|
||||
- dict with "content" key — o1/o3 thoughts/reasoning parts
|
||||
"""
|
||||
parts = content_obj.get("parts", [])
|
||||
if not parts:
|
||||
return ""
|
||||
@@ -680,16 +735,19 @@ def _extract_text(content_obj: dict, conv_id: str, node_id: str) -> str:
|
||||
if isinstance(part, str):
|
||||
text_parts.append(part)
|
||||
elif isinstance(part, dict):
|
||||
# Could be an image or file reference — skip and warn
|
||||
part_type = part.get("content_type", "unknown")
|
||||
if part_type != "text":
|
||||
part_type = part.get("content_type", "")
|
||||
if part_type == "text":
|
||||
text_parts.append(part.get("text", ""))
|
||||
elif "content" in part:
|
||||
# o1/o3 thoughts parts: {"summary": "...", "content": "..."}
|
||||
text_parts.append(part["content"])
|
||||
elif part_type:
|
||||
# Image, file, or other binary attachment — skip and warn
|
||||
logger.warning(
|
||||
"[chatgpt] Skipping %s attachment in conversation %s "
|
||||
"— rich content not yet supported (see FUTURE.md)",
|
||||
part_type,
|
||||
conv_id[:8],
|
||||
)
|
||||
else:
|
||||
text_parts.append(part.get("text", ""))
|
||||
|
||||
return "\n".join(t for t in text_parts if t)
|
||||
|
||||
@@ -39,7 +39,7 @@ class ClaudeProvider(BaseProvider):
|
||||
"init",
|
||||
RuntimeError(
|
||||
"CLAUDE_SESSION_KEY is not set. "
|
||||
"Run 'python -m src.main auth' to configure it."
|
||||
"Run 'ai-chat-exporter auth' to configure it."
|
||||
),
|
||||
)
|
||||
# Set sessionKey in the cookie jar
|
||||
@@ -60,7 +60,7 @@ class ClaudeProvider(BaseProvider):
|
||||
"Note: Claude session keys are opaque — a 401 is the only expiry signal. "
|
||||
"To refresh: open claude.ai in Chrome → F12 → Application → Cookies "
|
||||
"→ find 'sessionKey' → copy the value. "
|
||||
"Then run 'python -m src.main auth' or update CLAUDE_SESSION_KEY in .env."
|
||||
"Then run 'ai-chat-exporter auth' or update CLAUDE_SESSION_KEY in .env."
|
||||
)
|
||||
logger.error(msg)
|
||||
raise ProviderError(
|
||||
|
||||
@@ -0,0 +1,129 @@
|
||||
"""CLI-level tests using Click's CliRunner — no live API calls required."""
|
||||
|
||||
import pytest
|
||||
from click.testing import CliRunner
|
||||
|
||||
from src.cache import Cache
|
||||
from src.main import _filter_by_project, cli
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _filter_by_project (T-27)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestFilterByProject:
|
||||
"""Unit tests for the project filter logic used by export/list/joplin."""
|
||||
|
||||
# ChatGPT conversations use the _project_name annotation key
|
||||
def _chatgpt(self, conv_id, project_name):
|
||||
return {"id": conv_id, "_project_name": project_name}
|
||||
|
||||
# Claude conversations use the project dict key
|
||||
def _claude(self, conv_id, project_name):
|
||||
proj = {"name": project_name} if project_name else None
|
||||
return {"id": conv_id, "project": proj}
|
||||
|
||||
def test_none_filter_keeps_no_project_chatgpt(self):
|
||||
convs = [self._chatgpt("a", None), self._chatgpt("b", "Python Course")]
|
||||
result = _filter_by_project(convs, "none")
|
||||
assert len(result) == 1
|
||||
assert result[0]["id"] == "a"
|
||||
|
||||
def test_none_filter_keeps_no_project_claude(self):
|
||||
convs = [self._claude("a", None), self._claude("b", "Python Course")]
|
||||
result = _filter_by_project(convs, "none")
|
||||
assert len(result) == 1
|
||||
assert result[0]["id"] == "a"
|
||||
|
||||
def test_name_filter_case_insensitive(self):
|
||||
convs = [
|
||||
self._chatgpt("a", "Python Course"),
|
||||
self._chatgpt("b", "Java Course"),
|
||||
self._chatgpt("c", None),
|
||||
]
|
||||
result = _filter_by_project(convs, "PYTHON")
|
||||
assert len(result) == 1
|
||||
assert result[0]["id"] == "a"
|
||||
|
||||
def test_name_filter_substring_match(self):
|
||||
convs = [
|
||||
self._chatgpt("a", "Python Advanced Course"),
|
||||
self._chatgpt("b", "Python Basics"),
|
||||
self._chatgpt("c", "JavaScript"),
|
||||
]
|
||||
result = _filter_by_project(convs, "python")
|
||||
assert len(result) == 2
|
||||
assert {c["id"] for c in result} == {"a", "b"}
|
||||
|
||||
def test_no_matches_returns_empty(self):
|
||||
convs = [self._chatgpt("a", "Python Course"), self._chatgpt("b", None)]
|
||||
result = _filter_by_project(convs, "ruby")
|
||||
assert result == []
|
||||
|
||||
def test_none_filter_excludes_all_with_projects(self):
|
||||
convs = [self._chatgpt("a", "Project A"), self._chatgpt("b", "Project B")]
|
||||
result = _filter_by_project(convs, "none")
|
||||
assert result == []
|
||||
|
||||
def test_empty_string_project_treated_as_no_project(self):
|
||||
convs = [{"id": "a", "_project_name": ""}, {"id": "b", "_project_name": "Real"}]
|
||||
result = _filter_by_project(convs, "none")
|
||||
assert len(result) == 1
|
||||
assert result[0]["id"] == "a"
|
||||
|
||||
def test_claude_project_string_matched(self):
|
||||
# Claude can also have project as a plain string
|
||||
convs = [{"id": "a", "project": "python-course"}, {"id": "b", "project": None}]
|
||||
result = _filter_by_project(convs, "python")
|
||||
assert len(result) == 1
|
||||
assert result[0]["id"] == "a"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# export --since validation (T-25)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestExportSinceValidation:
|
||||
"""Test that --since with an invalid date exits cleanly with an error message."""
|
||||
|
||||
def _pre_populated_cache(self, tmp_path) -> Cache:
|
||||
"""Create a cache that passes the ToS gate and first-run doctor check."""
|
||||
cache = Cache(tmp_path)
|
||||
cache.acknowledge_tos()
|
||||
cache.mark_exported("chatgpt", "dummy-conv", {"updated_at": "2024-01-01T00:00:00Z"})
|
||||
return cache
|
||||
|
||||
def test_invalid_since_date_exits_with_error(self, tmp_path):
|
||||
self._pre_populated_cache(tmp_path)
|
||||
|
||||
runner = CliRunner(mix_stderr=True)
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["--no-log-file", "export", "--since", "notadate"],
|
||||
env={
|
||||
"CHATGPT_SESSION_TOKEN": "eyJtesttoken",
|
||||
"CACHE_DIR": str(tmp_path),
|
||||
"EXPORT_DIR": str(tmp_path / "exports"),
|
||||
},
|
||||
)
|
||||
assert result.exit_code == 1
|
||||
assert "Invalid --since date" in result.output
|
||||
assert "YYYY-MM-DD" in result.output
|
||||
|
||||
def test_valid_since_date_does_not_error(self, tmp_path):
|
||||
"""A valid date should not produce the invalid-date error (may fail later on API)."""
|
||||
self._pre_populated_cache(tmp_path)
|
||||
|
||||
runner = CliRunner(mix_stderr=True)
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["--no-log-file", "export", "--since", "2024-01-01"],
|
||||
env={
|
||||
"CHATGPT_SESSION_TOKEN": "eyJtesttoken",
|
||||
"CACHE_DIR": str(tmp_path),
|
||||
"EXPORT_DIR": str(tmp_path / "exports"),
|
||||
},
|
||||
)
|
||||
assert "Invalid --since date" not in result.output
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Tests for src/config.py — token validation logic (T-14)."""
|
||||
|
||||
import logging
|
||||
import time
|
||||
|
||||
import jwt
|
||||
import pytest
|
||||
|
||||
from src.config import _validate_chatgpt_token
|
||||
|
||||
|
||||
class TestValidateChatGPTToken:
|
||||
def test_expired_token_logs_warning(self, caplog):
|
||||
# T-14: expired JWT must produce a clear warning
|
||||
payload = {"exp": int(time.time()) - 3600} # expired 1 hour ago
|
||||
token = jwt.encode(payload, "secret", algorithm="HS256")
|
||||
with caplog.at_level(logging.WARNING, logger="src.config"):
|
||||
result = _validate_chatgpt_token(token)
|
||||
assert any("expired" in r.message.lower() for r in caplog.records)
|
||||
assert result is not None # still returns the expiry datetime
|
||||
|
||||
def test_expiring_within_24h_logs_warning(self, caplog):
|
||||
payload = {"exp": int(time.time()) + 3600} # expires in 1 hour
|
||||
token = jwt.encode(payload, "secret", algorithm="HS256")
|
||||
with caplog.at_level(logging.WARNING, logger="src.config"):
|
||||
_validate_chatgpt_token(token)
|
||||
assert any("less than 24 hours" in r.message for r in caplog.records)
|
||||
|
||||
def test_valid_token_no_expiry_warning(self, caplog):
|
||||
payload = {"exp": int(time.time()) + 86400 * 5} # valid for 5 days
|
||||
token = jwt.encode(payload, "secret", algorithm="HS256")
|
||||
with caplog.at_level(logging.WARNING, logger="src.config"):
|
||||
result = _validate_chatgpt_token(token)
|
||||
assert not any("expired" in r.message.lower() for r in caplog.records)
|
||||
assert result is not None
|
||||
|
||||
def test_token_without_exp_claim_logs_warning(self, caplog):
|
||||
payload = {"sub": "user123"} # no exp
|
||||
token = jwt.encode(payload, "secret", algorithm="HS256")
|
||||
with caplog.at_level(logging.WARNING, logger="src.config"):
|
||||
result = _validate_chatgpt_token(token)
|
||||
assert any("'exp'" in r.message or "no 'exp'" in r.message for r in caplog.records)
|
||||
assert result is None
|
||||
|
||||
def test_jwe_encrypted_token_returns_none(self, caplog):
|
||||
# JWE tokens (alg=dir) cannot be decoded client-side — this is normal for ChatGPT
|
||||
jwe_like = "eyJhbGciOiJkaXIiLCJlbmMiOiJBMjU2R0NNIn0.fake.token.data.here"
|
||||
with caplog.at_level(logging.DEBUG, logger="src.config"):
|
||||
result = _validate_chatgpt_token(jwe_like)
|
||||
assert result is None # cannot decode, but not an error
|
||||
|
||||
def test_non_jwt_string_logs_warning(self, caplog):
|
||||
with caplog.at_level(logging.WARNING, logger="src.config"):
|
||||
result = _validate_chatgpt_token("notajwttoken")
|
||||
assert any("does not look like a JWT" in r.message for r in caplog.records)
|
||||
assert result is None
|
||||
@@ -199,6 +199,34 @@ class TestJSONExporter:
|
||||
assert " " in raw
|
||||
|
||||
|
||||
class TestBothFormats:
|
||||
"""T-38: Markdown and JSON exporters produce matching filenames for the same conversation."""
|
||||
|
||||
def test_both_formats_produce_files(self, tmp_path):
|
||||
md_exp = MarkdownExporter(tmp_path)
|
||||
json_exp = JSONExporter(tmp_path)
|
||||
md_path = md_exp.export(SAMPLE_CONV)
|
||||
json_path = json_exp.export(SAMPLE_CONV)
|
||||
assert md_path.exists()
|
||||
assert json_path.exists()
|
||||
|
||||
def test_both_formats_have_matching_stems(self, tmp_path):
|
||||
md_exp = MarkdownExporter(tmp_path)
|
||||
json_exp = JSONExporter(tmp_path)
|
||||
md_path = md_exp.export(SAMPLE_CONV)
|
||||
json_path = json_exp.export(SAMPLE_CONV)
|
||||
assert md_path.suffix == ".md"
|
||||
assert json_path.suffix == ".json"
|
||||
assert md_path.stem == json_path.stem
|
||||
|
||||
def test_both_formats_same_directory(self, tmp_path):
|
||||
md_exp = MarkdownExporter(tmp_path)
|
||||
json_exp = JSONExporter(tmp_path)
|
||||
md_path = md_exp.export(SAMPLE_CONV)
|
||||
json_path = json_exp.export(SAMPLE_CONV)
|
||||
assert md_path.parent == json_path.parent
|
||||
|
||||
|
||||
class TestYamlEscape:
|
||||
def test_escapes_double_quotes(self):
|
||||
assert _yaml_escape('Say "hello"') == 'Say \\"hello\\"'
|
||||
|
||||
@@ -75,6 +75,39 @@ class TestChatGPTNormalization:
|
||||
for r in caplog.records
|
||||
)
|
||||
|
||||
def test_model_editable_context_included_without_warning(self, caplog):
|
||||
"""model_editable_context messages (project instructions) should be included, not warned about."""
|
||||
import logging
|
||||
conv = {
|
||||
"id": "test-conv-mec",
|
||||
"title": "Test",
|
||||
"create_time": 1700000000.0,
|
||||
"update_time": 1700000001.0,
|
||||
"mapping": {
|
||||
"root": {"id": "root", "message": None, "parent": None, "children": ["msg1"]},
|
||||
"msg1": {
|
||||
"id": "msg1",
|
||||
"message": {
|
||||
"id": "msg1",
|
||||
"author": {"role": "user"},
|
||||
"content": {
|
||||
"content_type": "model_editable_context",
|
||||
"parts": ["These are the project instructions."],
|
||||
},
|
||||
"create_time": 1700000001.0,
|
||||
"status": "finished_successfully",
|
||||
},
|
||||
"parent": "root",
|
||||
"children": [],
|
||||
},
|
||||
},
|
||||
}
|
||||
p = self._get_provider()
|
||||
with caplog.at_level(logging.WARNING):
|
||||
result = p.normalize_conversation(conv)
|
||||
assert any(m["content"] == "These are the project instructions." for m in result["messages"])
|
||||
assert not any("model_editable_context" in r.message for r in caplog.records)
|
||||
|
||||
def test_message_roles_are_valid(self):
|
||||
raw = json.loads((FIXTURES / "chatgpt_conversation.json").read_text())
|
||||
p = self._get_provider()
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Tests for src/utils.py — filename generation, path building, redaction."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from src.utils import (
|
||||
build_export_path,
|
||||
format_token_status,
|
||||
generate_filename,
|
||||
redact_secrets,
|
||||
)
|
||||
|
||||
|
||||
class TestGenerateFilename:
|
||||
def test_basic_format(self):
|
||||
name = generate_filename("Hello World", "abc12345def", "2024-06-10T14:00:00Z")
|
||||
assert name == "2024-06-10_hello-world_abc12345.md"
|
||||
|
||||
def test_special_chars_slugified(self):
|
||||
# T-36: titles with punctuation must produce safe, OS-compatible filenames
|
||||
name = generate_filename("What's this?! A test.", "abc12345", "2024-06-01T00:00:00Z")
|
||||
assert "?" not in name
|
||||
assert "!" not in name
|
||||
assert "'" not in name
|
||||
assert " " not in name
|
||||
assert name.startswith("2024-06-01_")
|
||||
assert name.endswith("_abc12345.md")
|
||||
|
||||
def test_unicode_chars_handled(self):
|
||||
name = generate_filename("Héllo Wörld", "abc12345", "2024-06-01T00:00:00Z")
|
||||
assert " " not in name
|
||||
assert name.endswith("_abc12345.md")
|
||||
|
||||
def test_empty_title_becomes_untitled(self):
|
||||
name = generate_filename("", "abc12345", "2024-06-01T00:00:00Z")
|
||||
assert "untitled" in name
|
||||
|
||||
def test_id_truncated_to_8_chars(self):
|
||||
name = generate_filename("Test", "abcdefghijklmnop", "2024-06-01T00:00:00Z")
|
||||
assert name.endswith("_abcdefgh.md")
|
||||
|
||||
def test_long_title_truncated(self):
|
||||
long_title = "a" * 200
|
||||
name = generate_filename(long_title, "abc12345", "2024-06-01T00:00:00Z")
|
||||
# Slug is capped at 60 chars by max_length
|
||||
slug_part = name.split("_")[1]
|
||||
assert len(slug_part) <= 60
|
||||
|
||||
def test_date_comes_from_created_at(self):
|
||||
name = generate_filename("Test", "abc12345", "2023-11-25T00:00:00Z")
|
||||
assert name.startswith("2023-11-25_")
|
||||
|
||||
|
||||
class TestBuildExportPath:
|
||||
def test_default_structure_provider_project_year(self):
|
||||
path = build_export_path(
|
||||
Path("/exports"), "claude", "my-project", "2024-06-01T00:00:00Z", "file.md"
|
||||
)
|
||||
assert str(path) == "/exports/claude/my-project/2024/file.md"
|
||||
|
||||
def test_no_project_uses_no_project_slug(self):
|
||||
path = build_export_path(
|
||||
Path("/exports"), "chatgpt", None, "2024-06-01T00:00:00Z", "file.md"
|
||||
)
|
||||
assert "no-project" in str(path)
|
||||
|
||||
def test_provider_project_structure_omits_year(self):
|
||||
path = build_export_path(
|
||||
Path("/exports"), "claude", "proj", "2024-06-01T00:00:00Z", "file.md",
|
||||
structure="provider/project",
|
||||
)
|
||||
assert "2024" not in str(path)
|
||||
assert "proj" in str(path)
|
||||
|
||||
def test_provider_year_structure_omits_project(self):
|
||||
path = build_export_path(
|
||||
Path("/exports"), "claude", "proj", "2024-06-01T00:00:00Z", "file.md",
|
||||
structure="provider/year",
|
||||
)
|
||||
assert "proj" not in str(path)
|
||||
assert "2024" in str(path)
|
||||
|
||||
def test_project_name_with_spaces_is_slugified(self):
|
||||
path = build_export_path(
|
||||
Path("/exports"), "claude", "My Project Name!", "2024-06-01T00:00:00Z", "file.md"
|
||||
)
|
||||
assert " " not in str(path)
|
||||
assert "!" not in str(path)
|
||||
|
||||
|
||||
class TestRedactSecrets:
|
||||
def test_token_value_redacted(self):
|
||||
data = {"token": "supersecret"}
|
||||
result = redact_secrets(data)
|
||||
assert result["token"] == "[REDACTED]"
|
||||
|
||||
def test_session_key_redacted(self):
|
||||
result = redact_secrets({"sessionKey": "abc123"})
|
||||
assert result["sessionKey"] == "[REDACTED]"
|
||||
|
||||
def test_non_sensitive_key_unchanged(self):
|
||||
result = redact_secrets({"title": "My Chat", "id": "abc123"})
|
||||
assert result["title"] == "My Chat"
|
||||
assert result["id"] == "abc123"
|
||||
|
||||
def test_nested_dict_redacted(self):
|
||||
data = {"user": {"token": "secret", "name": "Alice"}}
|
||||
result = redact_secrets(data)
|
||||
assert result["user"]["token"] == "[REDACTED]"
|
||||
assert result["user"]["name"] == "Alice"
|
||||
|
||||
def test_list_of_dicts(self):
|
||||
data = [{"password": "p@ss"}, {"title": "chat"}]
|
||||
result = redact_secrets(data)
|
||||
assert result[0]["password"] == "[REDACTED]"
|
||||
assert result[1]["title"] == "chat"
|
||||
|
||||
|
||||
class TestFormatTokenStatus:
|
||||
def test_none_token_returns_not_set(self):
|
||||
assert format_token_status(None) == "[NOT SET]"
|
||||
|
||||
def test_empty_token_returns_not_set(self):
|
||||
assert format_token_status("") == "[NOT SET]"
|
||||
|
||||
def test_set_token_no_expiry(self):
|
||||
assert format_token_status("sometoken") == "[SET]"
|
||||
|
||||
def test_expired_token(self):
|
||||
from datetime import datetime, timezone, timedelta
|
||||
expiry = datetime.now(tz=timezone.utc) - timedelta(days=1)
|
||||
result = format_token_status("tok", expiry)
|
||||
assert "EXPIRED" in result
|
||||
|
||||
def test_expiring_today_shows_hours(self):
|
||||
from datetime import datetime, timezone, timedelta
|
||||
expiry = datetime.now(tz=timezone.utc) + timedelta(hours=3)
|
||||
result = format_token_status("tok", expiry)
|
||||
assert "expires in" in result
|
||||
assert "h" in result
|
||||
|
||||
def test_expiring_in_days(self):
|
||||
from datetime import datetime, timezone, timedelta
|
||||
expiry = datetime.now(tz=timezone.utc) + timedelta(days=10, hours=12)
|
||||
result = format_token_status("tok", expiry)
|
||||
assert "10 days" in result
|
||||
Reference in New Issue
Block a user