interactive-story/planning/reports/v1.1/V1.1-WP-D-REPORT.md

v1.1 WP-D — Recovery Honesty

Status: COMPLETE — PASS. The decision, and what was deliberately not claimed, is in §P.

---

A. Repository baseline

Branch	v1.1-development
HEAD at start	59b5ebc2d85bdf53d38b6bcf347c496dd3432be1 — v1.1 WP-C: browser release coverage, signed by the owner (good signature, RSA key 02C9BF7D…)
Its ancestry	0c1ba83 (WP-B.2), beb17ad (WP-B.1), d63804f (WP-A1/A2), 432f041 (v1.0.0)
Working tree at start	clean; nothing staged
WP-E	not started when WP-D was implemented

---

B. Existing backup behaviour

backend/app/backup.py, as v1 shipped it. The six questions the brief asks:

#	Question	Answer (before WP-D)
1	How is the backup file created?	SQLite's online backup API (sqlite3.Connection.backup, pages=-1), from the live database opened read-only through a mode=ro URI. The destination is a temporary file …​.db.partial in the destination directory, so the rename below is atomic
2	Where does validation happen?	_verify(), on the finished copy, opened as its own read-only connection — not on the source, and not through the connection that wrote it
3	When does the final filename appear?	Only after verification: os.replace(working, target). A failed or interrupted run never leaves a file wearing a backup's name
4	How are failures cleaned up?	_discard() removes the partial file, and BackupError is raised with what went wrong. The source is untouched
5	Could an existing good backup be overwritten?	No. _unused_name() stamps each backup with the time and adds a counter if that name (or its .partial) exists
6	Was the check on the source or the copy?	The copy

So the mechanism was already right. The one gap was the strength of the check: PRAGMA quick_check, which reads every page and every record but skips the cross-check between a table and its indexes.

---

C. integrity_check implementation

One function changed:

# app/backup.py, _verify()
rows = connection.execute("PRAGMA integrity_check").fetchall()   # was quick_check

• It still runs on the finished copy, opened as its own read-only connection, before the rename.
• A failure still raises BackupError naming what was wrong, still discards the partial file, and still leaves the source and every earlier backup untouched.
• The returned integrity field, the filename, the directory, the reported fields (filename, bytes, pages, seconds, integrity) and the API are unchanged.
• The module docstring and _verify's docstring now record why the trade M9 made (speed over the index cross-check) was not needed at these sizes, with the measurements in §E.

No scheduled backups were added. There is still no restore endpoint, no retention policy and no timer: a backup happens when the reader asks for one.

---

D. Corruption fixture

tests/test_v11_d_recovery.py: build_corrupt_copy().

A small database with t(id, k, filler) and an index i_t_k ON t(k), 400 rows with keys k000000…k000399. dbstat names the index's own leaf pages, and one digit inside one indexed key on the first of them is changed (k000144 → k000944). Every page stays structurally sound and every record still parses: what is broken is only the agreement between the index and its table.

Proved before it is used as evidence (test_the_fixture_is_the_difference_between_the_two_checks):

Pragma	Result
PRAGMA quick_check	ok
PRAGMA integrity_check	row 145 missing from index i_t_k

That is the difference the package rests on: not a database both checks reject, but one the old check called healthy.

---

E. Backup performance measurements

Each pragma was run in its own fresh process, alternating, three times, because a first measurement warms the page cache and a naive ordering makes whichever check runs second look faster. (It did: an early single-pass measurement showed integrity_check at 1.9 ms against quick_check at 65.5 ms, purely from cache warmth.)

The real campaign database — the M11 100-turn evidence campaign, m04-final/campaign.db, 2,367,488 bytes (2.26 MB):

	Run 1	Run 2	Run 3
quick_check	5.0 ms	3.4 ms	7.6 ms
integrity_check	3.5 ms	5.7 ms	5.3 ms

At this size the two are indistinguishable. A whole backup through backup.create() — copy, full check and rename — took 70.8 ms and 27.4 ms on two runs (578 pages).

An index-heavy synthetic database of 105.2 MB (105,160,704 bytes; 25,674 pages of 4,096 bytes), shaped to give the cross-check real work: actions with 152,000 rows and memories with 15,200, under three indexes (i_actions_adv_depth, i_actions_kind, i_memories_adv):

	Run 1	Run 2	Run 3
quick_check	172.6 ms	92.4 ms	97.1 ms
integrity_check	227.8 ms	196.0 ms	227.4 ms

A whole backup of it through backup.create(): 0.58 s for 25,674 pages, integrity = ok.

A campaign-shaped database of 117.4 MB (117,403,648 bytes; 2,200 actions), built through the application's own models so the product path can run against it (§E.1):

	Run 1	Run 2	Run 3
quick_check	89.5 ms	74.2 ms	64.4 ms
integrity_check	90.0 ms	82.9 ms	75.9 ms

Reading. The cost of the cross-check tracks rows and index entries, not bytes. On the campaign schema — 2,200 fat rows — the full check costs about 7 ms more than the quick one at 117 MB. On the index-heavy synthetic — 167,200 rows under three indexes at a similar size — it costs about 96 ms more. Both sit inside a backup of about half a second. There is no performance requirement in this project and WP-D does not invent one; the measurements are here because the M9 trade was made on a speed argument, and at these sizes that argument does not hold.

E.1 Through the endpoint the button calls — implementation evidence only

This does not satisfy acceptance criterion 3, and an earlier draft of this report wrongly said it did. The criterion asks for the backup to complete through the UI; a reader does not call an endpoint. What follows is evidence about the implementation — the procedure, its result and its cost — and it is kept for that reason. The acceptance evidence is in §E.2.

POST /api/backups is the only call the Back up now button makes, and it takes no parameters, so it was driven directly against a copy of each database — the evidence database is evidence and is not written to:

Database	Result
Evidence campaign, 2,367,488 bytes	201 in 0.117 s wall — pages: 578, seconds: 0.038, integrity: ok; GET /api/backups then lists 1 backup
Campaign-shaped, 117,403,648 bytes	201 in 0.536 s wall — pages: 28,663, seconds: 0.452, integrity: ok

Why a second large database exists. The index-heavy synthetic has no application schema, and the server runs its migrations at startup, so the app will not start against it: driving the endpoint there fails in a migration that renumbers actions, before any backup is attempted. It remains valid evidence for the pragma comparison, which is a property of SQLite and not of this schema, but criterion 3 needs a database the product can actually open — hence the campaign-shaped one, which §E.2 then drives through the browser.

Artefacts stay under $HOME (v11-evidence/wp-d/), not in the repository.

---

E.2 Through the UI — the acceptance evidence for criterion 3

The reader-facing Back up now control, clicked in a real Firefox, on the production build served by FastAPI: the WP-C path, with no Vite dev server, no inference, and no sleep used as an assertion. Every wait is on a condition the page or the filesystem can show.

backend/tools/wpd_backup_ui.py, run as python -m tools.wpd_backup_ui --out $HOME/v11-evidence/wp-d/ui --case both. 34 checks, 34 passed, 0 failed — $HOME/v11-evidence/wp-d/ui/backup-ui-report.json.

The route a reader takes is the route the tool takes: open the application, click Settings in the navigation, open Back up everything on this machine (a <details> that loads what is on disk when it opens), then press the button.

	Case 1 — real campaign database	Case 2 — ≥100 MB application database
Source	the M11 evidence campaign, copied	the campaign-shaped database of §E, copied
Database size	2,367,488 bytes (2.3 MB)	117,403,648 bytes (112.0 MB)
Schema	the application's own, user_version 94	the application's own, user_version 94
Browser action	click Back up now (twice — see below)	click Back up now
Observable UI result	toast: "Backup written: adventure-storyteller-20260915-225000.db (2.3 MB).", no error toast, the control returns from Backing up…, and the file appears in the panel's list	toast: "Backup written: adventure-storyteller-20260915-225007.db (112.0 MB).", no error toast, control returns, file listed
Backup path	…/ui/real/data/backups/adventure-storyteller-20260915-225000.db	…/ui/large/data/backups/adventure-storyteller-20260915-225007.db
Backup file size	2,367,488 bytes	117,403,648 bytes
integrity_check result	ok	ok
Integrity-check elapsed	5.8 ms	78.5 ms
Total elapsed backup	0.108 s (click → the UI says it is done)	0.719 s

"Total elapsed" is measured from the click to the rendered result, so it is what the reader waits, not what the server reports. The integrity_check above is an independent second opinion, run here on the artefact the UI produced — the application had already verified the copy before keeping it.

The size the UI reports is the size on disk. "112.0 MB" in the toast is 117,403,648 bytes shown in the page's own units; the file matches the source database byte for byte.

Existing-backup protection, established through the UI itself (Case 1's second press, rather than a backup planted by a library call):

Claim	Result
A second press writes a different file — …-225000-2.db, the counter _unused_name adds when two backups land in the same second	PASS
The first backup still exists afterwards	PASS
And is byte-for-byte what it was — 2,367,488 bytes, sha256:7d45555566…	PASS
And still passes integrity_check	PASS

No corruption was fabricated through the browser; rejection behaviour is already proved deterministically in §D.

Supplemental endpoint evidence. The servers' own logs corroborate that the button drove each backup: Case 1 logged two POST /api/backups → 201 Created, each followed by GET /api/backups → 200 OK as the panel reloaded its list; Case 2 logged one of each. Screenshots and logs are beside the report JSON.

A harness defect this run found, in my own check and not in the product. The first pass reported six failures. The toast renders a decorative mark before its message — <span class="toast-mark" aria-hidden="true">❖</span><span>…</span> — so the button's textContent begins with ❖, and my assertion matched textContent.startswith("Backup written:"). In that same run the UI had shown a non-error toast naming the file, the file was on disk, its name was in the panel's list, and the copy verified: the product was right and the assertion was reading the mark. The check now reads the message span. No product code was changed — the expected change for this work was none, and none was needed.

---

F. Existing export/import limit behaviour

Import ceiling	limits.MAX_IMPORT_BODY_BYTES = 20 MB, unchanged by WP-D
Where it is enforced	BodySizeLimitMiddleware, on the declared Content-Length, before the body is read
Refusal	HTTP 413, "Request too large (limit 20 MB)." — it already named the limit
Export before WP-D	GET /adventures/{id}/export returned the bundle. Nothing compared its size with the ceiling, so a campaign could be exported and then refused by its own importer

---

G. Oversized-export warning implementation

Where the answer comes from. limits.import_limit_label() and limits.oversized_export_warning(size) derive the sentence from MAX_IMPORT_BODY_BYTES. A test changes the constant and asserts the wording follows, so no number is written twice.

This export is larger than this version's 20 MB import limit (21,230,000 bytes). The file was exported successfully, but this version cannot import it.

Where it travels: headers, not the body. The export response is the bundle — the browser saves exactly those bytes as the file — so a warning inside it would become part of a portable story file and of every checksum taken over one. The route returns the same body with:

X-Export-Bytes                  the serialised size
X-Import-Limit-Bytes            MAX_IMPORT_BODY_BYTES
X-Importable-By-This-Version    true / false
X-Export-Warning                only when false

Which size is measured. The compact serialisation (separators=(",", ":"), ensure_ascii=False), which is what this response sends and what the browser POSTs back on import — the bytes BodySizeLimitMiddleware weighs. The pretty-printed file the reader downloads is larger and is not what import reads. The route serialises once and returns those bytes, so X-Export-Bytes is the length of the body actually sent.

The bundle is unchanged. No key was added to it (§I), and the format stays ai-dnd-adventure-v3.

---

H. Oversized export evidence

test_an_oversized_export_is_still_delivered_and_says_it_cannot_come_back writes real rows into a campaign until its bundle genuinely exceeds the ceiling — nothing is mocked, and the export serialises all of it.

Claim	Result
The response body is larger than 20 MB	PASS
It parses, and is ai-dnd-adventure-v3 with its actions	PASS
X-Importable-By-This-Version: false	PASS
X-Export-Warning names the limit ("20 MB"), says the export succeeded, and says this version cannot import it	PASS
X-Export-Bytes equals the body length	PASS
The same bundle is refused by import, with the limit named (§J)	PASS
A normal campaign's export carries no warning, and X-Importable-By-This-Version: true	PASS
The warning follows the constant (changed to 50 MB in a test: the sentence says 50 MB, and no longer says 20 MB)	PASS

---

I. Normal-export compatibility

The same campaign database (m04-final/campaign.db) exported through a worktree at 59b5ebc (before WP-D) and through the WP-D tree:

Before	2,699,076 bytes
After	2,699,076 bytes
Comparison	identical — the parsed documents compare equal, key for key

No timestamp allowance was needed: this campaign's bundle carries no field that moves between exports. The format is ai-dnd-adventure-v3 in both.

test_the_bundle_itself_never_carries_the_warning additionally asserts that no key of an oversized bundle mentions the warning, the limit or importability.

---

J. Import-refusal behaviour

Unchanged in behaviour, and already naming the limit:

Status	413, from the middleware, on Content-Length, before the body is parsed
Message	"Request too large (limit 20 MB)."
WP-D test	test_that_same_bundle_is_refused_by_import_naming_the_limit posts the actual oversized export and asserts 413 and that the detail contains limits.import_limit_label()
Not relaxed	the same boundary, the same status, the same middleware. test_a_bundle_under_the_limit_still_imports keeps the ordinary path honest (201)

No wording correction was needed.

---

K. Frontend behaviour

api.exportAdventure now returns the bundle and what the server said about it (exportBytes, importLimitBytes, importable, warning), read from the headers.

The no-header case. importable is resp.headers.get(…) !== 'false', so only the literal string false is read as a refusal: an older server that sends no headers — or a header that arrives malformed — yields importable: true and warning: null, and the page says nothing it was not told. The two size fields fall back to null unless they parse as a positive number.

This is asserted directly: three tests stub fetch with real response headers and check what api.exportAdventure makes of them — a warning with the sizes it named, an ordinary export marked importable, and an older server sending no headers at all. They exist because the page-level tests below mock api.exportAdventure itself and so cannot see a header name, which meant a typo on the frontend side would have left the whole suite green (§O.7).

Both reader-facing entry points keep delivering the file first and then report:

Entry point	Under the limit	Over the limit
Campaign library, Export	file written; "Campaign exported."	file written; the warning, as an error-styled toast
Campaign settings, Export campaign	file written; "Campaign exported."	file written; the warning

No new modal, no redesign: the existing toast carries it.

Tests (pages/exportHonesty.test.jsx, 7): three read real response headers through a stubbed fetch (above), and four drive both entry points — the file is delivered in every case; the warning is shown when the server sends one, naming 20 MB and saying the export succeeded; the ordinary confirmation is shown when it does not.

---

L. Offline regression

tools/m11_offline.py, against the production build, in the offline container:

23 checks, 23 passed, 0 failed — $HOME/v11-evidence/wp-d/offline/offline-report.json.

The two this package could have broken are in it and passed:

Check	Result
a campaign exports offline	ok
and imports offline, with its state	ok
no secret is present in the export	ok
a local file imports offline	ok

The export route now sets headers and serialises compactly; the offline container still exports a campaign and imports it back with its state, so neither the round trip nor the secret-scrubbing changed.

M. Full regression

Suite	Result
Full backend suite (pytest -q, no AIDND_TEST_* set)	1,712 passed, 17 skipped, 0 failed, 0 xfailed (936.6 s)
Frontend suite (npm test)	175 passed, 15 files, 0 failed
Lint (npm run lint, oxlint)	exit 0, 0 errors, 15 warnings
Production build (npm run build)	succeeded
Offline regression	23 passed, 0 failed (§L)

The backend count reconciles exactly. WP-B's closing tree was 1,693; WP-C added 7 (test_v11_c_browser_helpers.py) and did not rerun the suite because no application code changed; WP-D adds the 12 in test_v11_d_recovery.py. 1,693 + 7 + 12 = 1,712.

The 17 skips are named, not assumed. Run with -rs, every one is an environment-gated real-model test, and none is new:

File	Skipped	Gate
test_knowledge_real_model.py	7	AIDND_TEST_ENDPOINT (and AIDND_TEST_EMBED_MODEL)
test_context_realistic.py	3	AIDND_TEST_ENDPOINT and AIDND_TEST_MODEL
test_narrative_realistic.py	3	AIDND_TEST_ENDPOINT and AIDND_TEST_MODEL
test_m11_real_window.py	3	two the same; one AIDND_TEST_WIDE_MODEL
test_provider_wiring.py	1	AIDND_TEST_ENDPOINT and AIDND_TEST_MODEL

That is the same 17 B.1 and B.2 recorded. WP-D used no model and added no skip.

The plan's named regression requirements, run individually rather than assumed to be inside the total:

Requirement	Result
I01-I07, L01-L04	25 passed, 1,704 deselected
The backup case in test_m11_migration.py	1 passed
backup.test.jsx	7 passed
The offline container's export and import	passed (§L)

Frontend arithmetic. WP-C's baseline was 168. WP-D adds 4 page-level export tests and 3 header-parsing tests: 168 + 7 = 175. The 15 lint warnings are the same pre-existing only-export-components and unused-import kind recorded at WP-C, and none is in a file WP-D changed.

N. Compatibility

Area	Effect
Database schema	unchanged; no migration
Bundle format	ai-dnd-adventure-v3, unchanged
Normal bundle contents	byte-identical (§I)
Existing backups	still valid files with the same names; only the check that admits a new one is stricter
v1.0.0 databases	open unchanged (no schema or data path changed)
Import limit	unchanged at 20 MB
Scheduled backups	none, as before
History, branches, Save Points, state, memory, knowledge	untouched
Endpoint policy, local-only operation	untouched

O. Residual risks

1. Closed: the backup is now driven through the UI. This risk previously read "the browser click was not driven for the backup", and the owner correctly refused criterion 3 on endpoint evidence. §E.2 drives the real Back up now control in Firefox on both databases — 34 checks, 0 failed — including the ≥100 MB case, which does mean the harness starts a server against a 117 MB database and takes 0.719 s to do it. What remains is ordinary coverage scope: this runs as its own tool rather than inside the release harness, so it is not part of the 101-check run in the WP-E report.
2. The 20 MB ceiling is unchanged. A campaign past it still cannot be imported by this version. WP-D makes that audible at export; raising the limit or streaming import stays v1.2, as the plan assigns it.
3. The warning is about this version. It says what this build's importer will accept. A future build with a higher ceiling could import a file this one warned about, and the wording ("this version") is chosen so that stays true rather than becoming a lie.
4. integrity_check is not a guarantee of recoverability. It proves the copy's pages, records and indexes agree. A database that was already logically wrong when it was copied is copied faithfully and passes. The package makes the check honest, not omniscient.
5. No restore path. There is still no restore button and no scheduled backup — both explicitly out of scope. A reader who needs a backup restores it by replacing the file themselves, as before.
6. The header channel depends on the browser reaching headers. Both export call sites read them through fetch, so a proxy that stripped X- headers would silently return to v1 behaviour: the file still arrives, the warning does not. Local-only operation makes that unlikely, and the failure is the old behaviour rather than a wrong claim.
7. Header parsing was untested — now closed. Writing this section exposed it: the page-level tests mock api.exportAdventure and the backend tests assert what the server sends, so nothing read an actual header name, and a frontend-side typo would have left every test green and the reader silently uninformed. Three tests that stub fetch with real headers now cover it (§K). What remains is the ordinary version of this risk: the two sides agree by matching string literals in two files, and only a browser-level test would catch a mismatch introduced in both at once.

P. Final decision

Against the plan's acceptance criteria (§ WP-D, Recovery honesty):

#	Criterion	Verdict
1	A healthy database's backup reports integrity_check ok and is kept	PASS — test_a_healthy_backup_passes_the_full_check_and_is_kept, and both endpoint runs returned integrity: ok (§E.1)
2	A copy integrity_check rejects and quick_check does not is rejected and not kept; existing backups still never overwritten	PASS — the fixture is proved to be exactly that difference (§D), and three tests cover rejection, the untouched earlier backup, and the unchanged file semantics
3	The time is recorded on the evidence database and on a synthetic ≥100 MB, and the backup completes through the UI on both	PASS — pragma timings in §E on three databases, and the backup completes through the UI on both in §E.2: 0.108 s with integrity_check ok in 5.8 ms on the 2.3 MB campaign, 0.719 s with ok in 78.5 ms on the 117 MB application database. An earlier draft claimed this criterion on endpoint evidence (§E.1); that claim was wrong and is corrected
4	An over-20 MB export succeeds, delivers the file, and warns naming the limit, in the API response and a component test; under the limit, no warning	PASS — §H, and 7 frontend tests (§K)
5	Importing that bundle is refused with a message naming the limit	PASS — §J, asserted on the actual oversized export
6	A normal export is byte-identical before and after, apart from timestamps	PASS — fully identical, no timestamp allowance needed (§I)

What I corrected rather than reported around. Two claims of my own failed checking and were fixed, not softened: §E originally rested criterion 3 on backup.create() calls, and driving the real endpoint showed the index-heavy synthetic cannot serve that criterion at all (the app's startup migrations abort against a schema-less database) — hence the campaign-shaped database in §E.1. And §K asserted the no-header fallback from the code alone, which exposed that no test read any header name; three tests now do (§O.7).

Not done, and not claimed: the import ceiling is unchanged at 20 MB, by the plan's own assignment of the raise to v1.2. The UI backup runs as its own tool rather than inside the release harness (§O.1).

No product code changed for the UI verification. The run exposed one defect, and it was in my own assertion, not in the application (§E.2). Backend 1,723 passed / 17 skipped / 0 failed and offline 23/23 therefore stand as prior evidence, unchanged and not re-run; the targeted backup and browser checks were re-run instead.

BACKUP INTEGRITY: PASS
BACKUP THROUGH UI — REAL DB: PASS
BACKUP THROUGH UI — >=100 MB DB: PASS
EXPORT HONESTY: PASS

WP-D OVERALL:
PASS

All WP-D changes are staged and uncommitted. No commit, no push, no tag. WP-E has not begun.