# v1.1 WP-D — Recovery Honesty **Status:** COMPLETE — **PASS**. The decision, and what was deliberately not claimed, is in §P. --- ## A. Repository baseline | | | | --- | --- | | Branch | `v1.1-development` | | HEAD at start | `59b5ebc2d85bdf53d38b6bcf347c496dd3432be1` — *v1.1 WP-C: browser release coverage*, signed by the owner (good signature, RSA key `02C9BF7D…`) | | Its ancestry | `0c1ba83` (WP-B.2), `beb17ad` (WP-B.1), `d63804f` (WP-A1/A2), `432f041` (v1.0.0) | | Working tree at start | clean; nothing staged | | WP-E | not started when WP-D was implemented | --- ## B. Existing backup behaviour `backend/app/backup.py`, as v1 shipped it. The six questions the brief asks: | # | Question | Answer (before WP-D) | | --- | --- | --- | | 1 | How is the backup file created? | SQLite's **online backup API** (`sqlite3.Connection.backup`, `pages=-1`), from the live database opened read-only through a `mode=ro` URI. The destination is a temporary file `…​.db.partial` **in the destination directory**, so the rename below is atomic | | 2 | Where does validation happen? | `_verify()`, on the finished copy, opened as its own read-only connection — not on the source, and not through the connection that wrote it | | 3 | When does the final filename appear? | Only after verification: `os.replace(working, target)`. A failed or interrupted run never leaves a file wearing a backup's name | | 4 | How are failures cleaned up? | `_discard()` removes the partial file, and `BackupError` is raised with what went wrong. The source is untouched | | 5 | Could an existing good backup be overwritten? | **No.** `_unused_name()` stamps each backup with the time and adds a counter if that name (or its `.partial`) exists | | 6 | Was the check on the source or the copy? | The copy | So the mechanism was already right. The one gap was the **strength** of the check: `PRAGMA quick_check`, which reads every page and every record but skips the cross-check between a table and its indexes. --- ## C. `integrity_check` implementation One function changed: ```python # app/backup.py, _verify() rows = connection.execute("PRAGMA integrity_check").fetchall() # was quick_check ``` - It still runs on the **finished copy**, opened as its own read-only connection, before the rename. - A failure still raises `BackupError` naming what was wrong, still discards the partial file, and still leaves the source and every earlier backup untouched. - The returned `integrity` field, the filename, the directory, the reported fields (`filename`, `bytes`, `pages`, `seconds`, `integrity`) and the API are unchanged. - The module docstring and `_verify`'s docstring now record why the trade M9 made (speed over the index cross-check) was not needed at these sizes, with the measurements in §E. **No scheduled backups were added.** There is still no restore endpoint, no retention policy and no timer: a backup happens when the reader asks for one. --- ## D. Corruption fixture `tests/test_v11_d_recovery.py: build_corrupt_copy()`. A small database with `t(id, k, filler)` and an index `i_t_k ON t(k)`, 400 rows with keys `k000000`…`k000399`. `dbstat` names the index's own leaf pages, and one digit inside one indexed key on the first of them is changed (`k000144` → `k000944`). Every page stays structurally sound and every record still parses: what is broken is only the agreement between the index and its table. **Proved before it is used as evidence** (`test_the_fixture_is_the_difference_between_the_two_checks`): | Pragma | Result | | --- | --- | | `PRAGMA quick_check` | **`ok`** | | `PRAGMA integrity_check` | **`row 145 missing from index i_t_k`** | That is the difference the package rests on: not a database both checks reject, but one the old check called healthy. --- ## E. Backup performance measurements Each pragma was run in its **own fresh process**, alternating, three times, because a first measurement warms the page cache and a naive ordering makes whichever check runs second look faster. (It did: an early single-pass measurement showed `integrity_check` at 1.9 ms against `quick_check` at 65.5 ms, purely from cache warmth.) **The real campaign database** — the M11 100-turn evidence campaign, `m04-final/campaign.db`, 2,367,488 bytes (2.26 MB): | | Run 1 | Run 2 | Run 3 | | --- | --- | --- | --- | | `quick_check` | 5.0 ms | 3.4 ms | 7.6 ms | | `integrity_check` | 3.5 ms | 5.7 ms | 5.3 ms | At this size the two are indistinguishable. A whole backup through `backup.create()` — copy, full check and rename — took **70.8 ms** and **27.4 ms** on two runs (578 pages). **An index-heavy synthetic database of 105.2 MB** (105,160,704 bytes; 25,674 pages of 4,096 bytes), shaped to give the cross-check real work: `actions` with 152,000 rows and `memories` with 15,200, under three indexes (`i_actions_adv_depth`, `i_actions_kind`, `i_memories_adv`): | | Run 1 | Run 2 | Run 3 | | --- | --- | --- | --- | | `quick_check` | 172.6 ms | 92.4 ms | 97.1 ms | | `integrity_check` | 227.8 ms | 196.0 ms | 227.4 ms | A whole backup of it through `backup.create()`: **0.58 s** for 25,674 pages, `integrity = ok`. **A campaign-shaped database of 117.4 MB** (117,403,648 bytes; 2,200 actions), built through the application's own models so the product path can run against it (§E.1): | | Run 1 | Run 2 | Run 3 | | --- | --- | --- | --- | | `quick_check` | 89.5 ms | 74.2 ms | 64.4 ms | | `integrity_check` | 90.0 ms | 82.9 ms | 75.9 ms | **Reading.** The cost of the cross-check tracks **rows and index entries, not bytes**. On the campaign schema — 2,200 fat rows — the full check costs about 7 ms more than the quick one at 117 MB. On the index-heavy synthetic — 167,200 rows under three indexes at a similar size — it costs about 96 ms more. Both sit inside a backup of about half a second. There is no performance requirement in this project and WP-D does not invent one; the measurements are here because the M9 trade was made on a speed argument, and at these sizes that argument does not hold. ### E.1 Through the endpoint the button calls — implementation evidence only **This does not satisfy acceptance criterion 3, and an earlier draft of this report wrongly said it did.** The criterion asks for the backup to *complete through the UI*; a reader does not call an endpoint. What follows is evidence about the implementation — the procedure, its result and its cost — and it is kept for that reason. The acceptance evidence is in **§E.2**. `POST /api/backups` is the only call the **Back up now** button makes, and it takes no parameters, so it was driven directly against a **copy** of each database — the evidence database is evidence and is not written to: | Database | Result | | --- | --- | | Evidence campaign, 2,367,488 bytes | **201** in 0.117 s wall — `pages: 578`, `seconds: 0.038`, `integrity: ok`; `GET /api/backups` then lists 1 backup | | Campaign-shaped, 117,403,648 bytes | **201** in 0.536 s wall — `pages: 28,663`, `seconds: 0.452`, `integrity: ok` | **Why a second large database exists.** The index-heavy synthetic has no application schema, and the server runs its migrations at startup, so the app will not start against it: driving the endpoint there fails in a migration that renumbers `actions`, before any backup is attempted. It remains valid evidence for the pragma comparison, which is a property of SQLite and not of this schema, but criterion 3 needs a database the product can actually open — hence the campaign-shaped one, which §E.2 then drives through the browser. Artefacts stay under `$HOME` (`v11-evidence/wp-d/`), not in the repository. --- ### E.2 Through the UI — the acceptance evidence for criterion 3 The reader-facing **Back up now** control, clicked in a real Firefox, on the production build served by FastAPI: the WP-C path, with no Vite dev server, no inference, and no sleep used as an assertion. Every wait is on a condition the page or the filesystem can show. `backend/tools/wpd_backup_ui.py`, run as `python -m tools.wpd_backup_ui --out $HOME/v11-evidence/wp-d/ui --case both`. **34 checks, 34 passed, 0 failed** — `$HOME/v11-evidence/wp-d/ui/backup-ui-report.json`. The route a reader takes is the route the tool takes: open the application, click **Settings** in the navigation, open *Back up everything on this machine* (a `
` that loads what is on disk when it opens), then press the button. | | **Case 1 — real campaign database** | **Case 2 — ≥100 MB application database** | | --- | --- | --- | | Source | the M11 evidence campaign, copied | the campaign-shaped database of §E, copied | | **Database size** | **2,367,488 bytes** (2.3 MB) | **117,403,648 bytes** (112.0 MB) | | Schema | the application's own, `user_version` 94 | the application's own, `user_version` 94 | | **Browser action** | click **Back up now** (twice — see below) | click **Back up now** | | **Observable UI result** | toast: *"Backup written: adventure-storyteller-20260915-225000.db (2.3 MB)."*, no error toast, the control returns from *Backing up…*, and the file appears in the panel's list | toast: *"Backup written: adventure-storyteller-20260915-225007.db (112.0 MB)."*, no error toast, control returns, file listed | | **Backup path** | `…/ui/real/data/backups/adventure-storyteller-20260915-225000.db` | `…/ui/large/data/backups/adventure-storyteller-20260915-225007.db` | | **Backup file size** | **2,367,488 bytes** | **117,403,648 bytes** | | **`integrity_check` result** | **`ok`** | **`ok`** | | **Integrity-check elapsed** | **5.8 ms** | **78.5 ms** | | **Total elapsed backup** | **0.108 s** (click → the UI says it is done) | **0.719 s** | "Total elapsed" is measured from the click to the rendered result, so it is what the reader waits, not what the server reports. The `integrity_check` above is an independent second opinion, run here on the artefact the UI produced — the application had already verified the copy before keeping it. **The size the UI reports is the size on disk.** "112.0 MB" in the toast is 117,403,648 bytes shown in the page's own units; the file matches the source database byte for byte. **Existing-backup protection, established through the UI itself** (Case 1's second press, rather than a backup planted by a library call): | Claim | Result | | --- | --- | | A second press writes a *different* file — `…-225000-2.db`, the counter `_unused_name` adds when two backups land in the same second | PASS | | The first backup still exists afterwards | PASS | | And is byte-for-byte what it was — 2,367,488 bytes, `sha256:7d45555566…` | PASS | | And still passes `integrity_check` | PASS | No corruption was fabricated through the browser; rejection behaviour is already proved deterministically in §D. **Supplemental endpoint evidence.** The servers' own logs corroborate that the button drove each backup: Case 1 logged two `POST /api/backups → 201 Created`, each followed by `GET /api/backups → 200 OK` as the panel reloaded its list; Case 2 logged one of each. Screenshots and logs are beside the report JSON. **A harness defect this run found, in my own check and not in the product.** The first pass reported six failures. The toast renders a decorative mark before its message — `…` — so the button's `textContent` begins with `❖`, and my assertion matched `textContent.startswith("Backup written:")`. In that same run the UI had shown a non-error toast naming the file, the file was on disk, its name was in the panel's list, and the copy verified: the product was right and the assertion was reading the mark. The check now reads the message span. **No product code was changed** — the expected change for this work was none, and none was needed. --- ## F. Existing export/import limit behaviour | | | | --- | --- | | Import ceiling | `limits.MAX_IMPORT_BODY_BYTES` = 20 MB, **unchanged by WP-D** | | Where it is enforced | `BodySizeLimitMiddleware`, on the declared `Content-Length`, before the body is read | | Refusal | HTTP **413**, "Request too large (limit 20 MB)." — it already named the limit | | Export before WP-D | `GET /adventures/{id}/export` returned the bundle. Nothing compared its size with the ceiling, so a campaign could be exported and then refused by its own importer | --- ## G. Oversized-export warning implementation **Where the answer comes from.** `limits.import_limit_label()` and `limits.oversized_export_warning(size)` derive the sentence from `MAX_IMPORT_BODY_BYTES`. A test changes the constant and asserts the wording follows, so no number is written twice. > This export is larger than this version's 20 MB import limit (21,230,000 bytes). > The file was exported successfully, but this version cannot import it. **Where it travels: headers, not the body.** The export response *is* the bundle — the browser saves exactly those bytes as the file — so a warning inside it would become part of a portable story file and of every checksum taken over one. The route returns the same body with: ```text X-Export-Bytes the serialised size X-Import-Limit-Bytes MAX_IMPORT_BODY_BYTES X-Importable-By-This-Version true / false X-Export-Warning only when false ``` **Which size is measured.** The compact serialisation (`separators=(",", ":")`, `ensure_ascii=False`), which is what this response sends *and* what the browser POSTs back on import — the bytes `BodySizeLimitMiddleware` weighs. The pretty-printed file the reader downloads is larger and is not what import reads. The route serialises once and returns those bytes, so `X-Export-Bytes` is the length of the body actually sent. **The bundle is unchanged.** No key was added to it (§I), and the format stays `ai-dnd-adventure-v3`. --- ## H. Oversized export evidence `test_an_oversized_export_is_still_delivered_and_says_it_cannot_come_back` writes real rows into a campaign until its bundle genuinely exceeds the ceiling — nothing is mocked, and the export serialises all of it. | Claim | Result | | --- | --- | | The response body is larger than 20 MB | PASS | | It parses, and is `ai-dnd-adventure-v3` with its actions | PASS | | `X-Importable-By-This-Version: false` | PASS | | `X-Export-Warning` names the limit ("20 MB"), says the export succeeded, and says this version cannot import it | PASS | | `X-Export-Bytes` equals the body length | PASS | | The same bundle is refused by import, with the limit named (§J) | PASS | | A normal campaign's export carries **no** warning, and `X-Importable-By-This-Version: true` | PASS | | The warning follows the constant (changed to 50 MB in a test: the sentence says 50 MB, and no longer says 20 MB) | PASS | --- ## I. Normal-export compatibility The same campaign database (`m04-final/campaign.db`) exported through a worktree at `59b5ebc` (before WP-D) and through the WP-D tree: | | | | --- | --- | | Before | 2,699,076 bytes | | After | 2,699,076 bytes | | Comparison | **identical** — the parsed documents compare equal, key for key | No timestamp allowance was needed: this campaign's bundle carries no field that moves between exports. The format is `ai-dnd-adventure-v3` in both. `test_the_bundle_itself_never_carries_the_warning` additionally asserts that no key of an oversized bundle mentions the warning, the limit or importability. --- ## J. Import-refusal behaviour Unchanged in behaviour, and already naming the limit: | | | | --- | --- | | Status | **413**, from the middleware, on `Content-Length`, before the body is parsed | | Message | "Request too large (limit 20 MB)." | | WP-D test | `test_that_same_bundle_is_refused_by_import_naming_the_limit` posts the *actual oversized export* and asserts 413 and that the detail contains `limits.import_limit_label()` | | Not relaxed | the same boundary, the same status, the same middleware. `test_a_bundle_under_the_limit_still_imports` keeps the ordinary path honest (201) | No wording correction was needed. --- ## K. Frontend behaviour `api.exportAdventure` now returns the bundle **and** what the server said about it (`exportBytes`, `importLimitBytes`, `importable`, `warning`), read from the headers. **The no-header case.** `importable` is `resp.headers.get(…) !== 'false'`, so only the literal string `false` is read as a refusal: an older server that sends no headers — or a header that arrives malformed — yields `importable: true` and `warning: null`, and the page says nothing it was not told. The two size fields fall back to `null` unless they parse as a positive number. This is asserted directly: three tests stub `fetch` with real response headers and check what `api.exportAdventure` makes of them — a warning with the sizes it named, an ordinary export marked importable, and an older server sending no headers at all. They exist because the page-level tests below mock `api.exportAdventure` itself and so cannot see a header name, which meant a typo on the frontend side would have left the whole suite green (§O.7). Both reader-facing entry points keep delivering the file first and then report: | Entry point | Under the limit | Over the limit | | --- | --- | --- | | Campaign library, **Export** | file written; "Campaign exported." | file written; the warning, as an error-styled toast | | Campaign settings, **Export campaign** | file written; "Campaign exported." | file written; the warning | No new modal, no redesign: the existing toast carries it. **Tests** (`pages/exportHonesty.test.jsx`, **7**): three read real response headers through a stubbed `fetch` (above), and four drive both entry points — the file is delivered in every case; the warning is shown when the server sends one, naming 20 MB and saying the export succeeded; the ordinary confirmation is shown when it does not. --- ## L. Offline regression `tools/m11_offline.py`, against the production build, in the offline container: **23 checks, 23 passed, 0 failed** — `$HOME/v11-evidence/wp-d/offline/offline-report.json`. The two this package could have broken are in it and passed: | Check | Result | | --- | --- | | a campaign exports offline | ok | | and imports offline, with its state | ok | | no secret is present in the export | ok | | a local file imports offline | ok | The export route now sets headers and serialises compactly; the offline container still exports a campaign and imports it back with its state, so neither the round trip nor the secret-scrubbing changed. ## M. Full regression | Suite | Result | | --- | --- | | **Full backend suite** (`pytest -q`, no `AIDND_TEST_*` set) | **1,712 passed, 17 skipped, 0 failed, 0 xfailed** (936.6 s) | | **Frontend suite** (`npm test`) | **175 passed**, 15 files, 0 failed | | **Lint** (`npm run lint`, oxlint) | **exit 0**, 0 errors, 15 warnings | | **Production build** (`npm run build`) | succeeded | | **Offline regression** | 23 passed, 0 failed (§L) | **The backend count reconciles exactly.** WP-B's closing tree was 1,693; WP-C added 7 (`test_v11_c_browser_helpers.py`) and did not rerun the suite because no application code changed; WP-D adds the 12 in `test_v11_d_recovery.py`. 1,693 + 7 + 12 = **1,712**. **The 17 skips are named, not assumed.** Run with `-rs`, every one is an environment-gated real-model test, and none is new: | File | Skipped | Gate | | --- | --- | --- | | `test_knowledge_real_model.py` | 7 | `AIDND_TEST_ENDPOINT` (and `AIDND_TEST_EMBED_MODEL`) | | `test_context_realistic.py` | 3 | `AIDND_TEST_ENDPOINT` and `AIDND_TEST_MODEL` | | `test_narrative_realistic.py` | 3 | `AIDND_TEST_ENDPOINT` and `AIDND_TEST_MODEL` | | `test_m11_real_window.py` | 3 | two the same; one `AIDND_TEST_WIDE_MODEL` | | `test_provider_wiring.py` | 1 | `AIDND_TEST_ENDPOINT` and `AIDND_TEST_MODEL` | That is the same 17 B.1 and B.2 recorded. WP-D used no model and added no skip. **The plan's named regression requirements**, run individually rather than assumed to be inside the total: | Requirement | Result | | --- | --- | | I01-I07, L01-L04 | **25 passed**, 1,704 deselected | | The backup case in `test_m11_migration.py` | **1 passed** | | `backup.test.jsx` | **7 passed** | | The offline container's export and import | passed (§L) | **Frontend arithmetic.** WP-C's baseline was 168. WP-D adds 4 page-level export tests and 3 header-parsing tests: 168 + 7 = **175**. The 15 lint warnings are the same pre-existing `only-export-components` and unused-import kind recorded at WP-C, and **none is in a file WP-D changed**. ## N. Compatibility | Area | Effect | | --- | --- | | Database schema | unchanged; no migration | | Bundle format | `ai-dnd-adventure-v3`, unchanged | | Normal bundle contents | byte-identical (§I) | | Existing backups | still valid files with the same names; only the check that admits a new one is stricter | | v1.0.0 databases | open unchanged (no schema or data path changed) | | Import limit | unchanged at 20 MB | | Scheduled backups | none, as before | | History, branches, Save Points, state, memory, knowledge | untouched | | Endpoint policy, local-only operation | untouched | ## O. Residual risks 1. **Closed: the backup is now driven through the UI.** This risk previously read "the browser click was not driven for the backup", and the owner correctly refused criterion 3 on endpoint evidence. §E.2 drives the real **Back up now** control in Firefox on both databases — 34 checks, 0 failed — including the ≥100 MB case, which does mean the harness starts a server against a 117 MB database and takes 0.719 s to do it. What remains is ordinary coverage scope: this runs as its own tool rather than inside the release harness, so it is not part of the 101-check run in the WP-E report. 2. **The 20 MB ceiling is unchanged.** A campaign past it still cannot be imported by this version. WP-D makes that audible at export; raising the limit or streaming import stays v1.2, as the plan assigns it. 3. **The warning is about *this* version.** It says what this build's importer will accept. A future build with a higher ceiling could import a file this one warned about, and the wording ("this version") is chosen so that stays true rather than becoming a lie. 4. **`integrity_check` is not a guarantee of recoverability.** It proves the copy's pages, records and indexes agree. A database that was already logically wrong when it was copied is copied faithfully and passes. The package makes the check honest, not omniscient. 5. **No restore path.** There is still no restore button and no scheduled backup — both explicitly out of scope. A reader who needs a backup restores it by replacing the file themselves, as before. 6. **The header channel depends on the browser reaching headers.** Both export call sites read them through `fetch`, so a proxy that stripped `X-` headers would silently return to v1 behaviour: the file still arrives, the warning does not. Local-only operation makes that unlikely, and the failure is the old behaviour rather than a wrong claim. 7. **Header parsing was untested — now closed.** Writing this section exposed it: the page-level tests mock `api.exportAdventure` and the backend tests assert what the server sends, so nothing read an actual header name, and a frontend-side typo would have left every test green and the reader silently uninformed. Three tests that stub `fetch` with real headers now cover it (§K). What remains is the ordinary version of this risk: the two sides agree by matching string literals in two files, and only a browser-level test would catch a mismatch introduced in both at once. ## P. Final decision **Against the plan's acceptance criteria** (§ WP-D, *Recovery honesty*): | # | Criterion | Verdict | | --- | --- | --- | | 1 | A healthy database's backup reports `integrity_check` ok and is kept | **PASS** — `test_a_healthy_backup_passes_the_full_check_and_is_kept`, and both endpoint runs returned `integrity: ok` (§E.1) | | 2 | A copy `integrity_check` rejects and `quick_check` does not is rejected and not kept; existing backups still never overwritten | **PASS** — the fixture is proved to be exactly that difference (§D), and three tests cover rejection, the untouched earlier backup, and the unchanged file semantics | | 3 | The time is recorded on the evidence database and on a synthetic ≥100 MB, and the backup completes through the UI on both | **PASS** — pragma timings in §E on three databases, and the backup completes **through the UI** on both in §E.2: 0.108 s with `integrity_check` `ok` in 5.8 ms on the 2.3 MB campaign, 0.719 s with `ok` in 78.5 ms on the 117 MB application database. An earlier draft claimed this criterion on endpoint evidence (§E.1); that claim was wrong and is corrected | | 4 | An over-20 MB export succeeds, delivers the file, and warns naming the limit, in the API response and a component test; under the limit, no warning | **PASS** — §H, and 7 frontend tests (§K) | | 5 | Importing that bundle is refused with a message naming the limit | **PASS** — §J, asserted on the actual oversized export | | 6 | A normal export is byte-identical before and after, apart from timestamps | **PASS** — fully identical, no timestamp allowance needed (§I) | **What I corrected rather than reported around.** Two claims of my own failed checking and were fixed, not softened: §E originally rested criterion 3 on `backup.create()` calls, and driving the real endpoint showed the index-heavy synthetic cannot serve that criterion at all (the app's startup migrations abort against a schema-less database) — hence the campaign-shaped database in §E.1. And §K asserted the no-header fallback from the code alone, which exposed that **no test read any header name**; three tests now do (§O.7). **Not done, and not claimed:** the import ceiling is unchanged at 20 MB, by the plan's own assignment of the raise to v1.2. The UI backup runs as its own tool rather than inside the release harness (§O.1). **No product code changed for the UI verification.** The run exposed one defect, and it was in my own assertion, not in the application (§E.2). Backend **1,723 passed / 17 skipped / 0 failed** and offline **23/23** therefore stand as prior evidence, unchanged and not re-run; the targeted backup and browser checks were re-run instead. ```text BACKUP INTEGRITY: PASS BACKUP THROUGH UI — REAL DB: PASS BACKUP THROUGH UI — >=100 MB DB: PASS EXPORT HONESTY: PASS WP-D OVERALL: PASS ``` All WP-D changes are **staged and uncommitted**. No commit, no push, no tag. WP-E has not begun.