v1.1: harden recovery and control boundaries
WP-D and WP-E complete the planned v1.1 implementation packages. WP-D — recovery honesty: - backups verify the completed copy with PRAGMA integrity_check - corruption missed by quick_check is detected by the full check - existing good backups remain protected - oversized exports are still delivered but declare whether this version can import them, while the 20 MB import limit remains unchanged - backup was exercised through the real browser UI on both the normal campaign database and a campaign-shaped database over 100 MB WP-E — control-boundary contrast: - interactive control boundaries meet the WCAG 1.4.11 3:1 target - the contrast audit is now a failing gate rather than an advisory - rendered browser measurements pass for the composer, controls, tabs and nav - text contrast and focus visibility remain intact - owner reviewed and approved the before/after screenshots Reports: - planning/reports/v1.1/V1.1-WP-D-REPORT.md - planning/reports/v1.1/V1.1-WP-E-REPORT.md All planned v1.1 work packages A-E are now complete. Release validation has not yet begun.
This commit is contained in:
JesseMarkowitz
+20
-9
@@ -39,8 +39,9 @@ turn is blocked.
|
||||
never leaves a half-written file wearing a backup's name. `os.replace` is
|
||||
atomic on the same filesystem, which is why the temporary sits in the
|
||||
destination's own directory rather than in `/tmp`.
|
||||
3. `PRAGMA quick_check` runs against the finished copy, opened as its own
|
||||
database, before it is renamed. A backup nobody verified is a belief.
|
||||
3. `PRAGMA integrity_check` runs against the finished copy, opened as its own
|
||||
database, before it is renamed. A backup nobody verified is a belief. v1.1
|
||||
WP-D made this the full check rather than `quick_check`; see `_verify`.
|
||||
4. An existing file is never overwritten. Each run writes a new name stamped
|
||||
with the time, so yesterday's backup survives today's mistake — which is most
|
||||
of what a backup is for.
|
||||
@@ -189,18 +190,28 @@ def _copy(source_path: Path, working: Path) -> int:
|
||||
|
||||
|
||||
def _verify(working: Path) -> str:
|
||||
"""Runs `PRAGMA quick_check` against the finished copy.
|
||||
"""Runs `PRAGMA integrity_check` against the finished copy.
|
||||
|
||||
Opened as its own connection, so what is checked is the file on disk rather
|
||||
than any page cache the copy left behind. `quick_check` rather than
|
||||
`integrity_check` because it does the structural work — every page reachable,
|
||||
every record readable — without the full index cross-check, which on a large
|
||||
database is minutes rather than moments. A backup nobody verified is a
|
||||
belief; a backup verified slowly enough that nobody takes one is worse.
|
||||
than any page cache the copy left behind.
|
||||
|
||||
**v1.1 WP-D: the full check, not `quick_check`.** M9 chose `quick_check` for
|
||||
its speed, on the argument that a backup verified slowly enough that nobody
|
||||
takes one is worse than a fast one. The measurements say the trade was not
|
||||
needed here: `quick_check` omits the cross-check between a table and its
|
||||
indexes, and that is a real class of damage it reports as `ok`. A copy whose
|
||||
index disagrees with its table restores into a database that answers queries
|
||||
with rows that are not there — the failure a backup exists to prevent.
|
||||
|
||||
The cost is small at the sizes this application produces: on the 100-turn
|
||||
evidence campaign both checks are a few milliseconds, and on a synthetic
|
||||
database two orders of magnitude larger the difference is still short of a
|
||||
second (WP-D report §E). A backup nobody verified is a belief; this is the
|
||||
check that makes it a fact.
|
||||
"""
|
||||
connection = sqlite3.connect(f"file:{working}?mode=ro", uri=True)
|
||||
try:
|
||||
rows = connection.execute("PRAGMA quick_check").fetchall()
|
||||
rows = connection.execute("PRAGMA integrity_check").fetchall()
|
||||
finally:
|
||||
connection.close()
|
||||
result = ", ".join(str(row[0]) for row in rows) if rows else "no result"
|
||||
|
||||
@@ -129,6 +129,34 @@ MAX_BODY_BYTES = 2 * 1024 * 1024
|
||||
MAX_IMPORT_BODY_BYTES = 20 * 1024 * 1024
|
||||
|
||||
|
||||
def import_limit_label(limit: int | None = None) -> str:
|
||||
"""The import ceiling as a reader would say it, e.g. "20 MB".
|
||||
|
||||
Derived from the constant rather than written beside it, so the refusal, the
|
||||
export warning and the documentation cannot drift apart from each other or
|
||||
from what the middleware actually enforces (v1.1 WP-D).
|
||||
"""
|
||||
size = MAX_IMPORT_BODY_BYTES if limit is None else limit
|
||||
megabytes = size / (1024 * 1024)
|
||||
return f"{megabytes:.0f} MB" if abs(megabytes - round(megabytes)) < 0.05 else f"{megabytes:.1f} MB"
|
||||
|
||||
|
||||
def oversized_export_warning(export_bytes: int, limit: int | None = None) -> str:
|
||||
"""What to tell a reader whose export is larger than import will accept.
|
||||
|
||||
v1.1 WP-D. The file is written and is not damaged: what it exceeds is this
|
||||
version's import ceiling, so it cannot be brought back in *here*. Saying that
|
||||
plainly is the whole point — the alternative is a reader who finds out when
|
||||
they try to restore it.
|
||||
"""
|
||||
size = MAX_IMPORT_BODY_BYTES if limit is None else limit
|
||||
return (
|
||||
f"This export is larger than this version's {import_limit_label(size)} import "
|
||||
f"limit ({export_bytes:,} bytes). The file was exported successfully, but this "
|
||||
f"version cannot import it."
|
||||
)
|
||||
|
||||
|
||||
class BodySizeLimitMiddleware:
|
||||
"""Rejects oversized request bodies by their declared `Content-Length`.
|
||||
|
||||
|
||||
@@ -28,7 +28,9 @@ repair. Refusing a whole campaign because a search index would not build would
|
||||
trade the valuable thing for the cheap one.
|
||||
"""
|
||||
|
||||
from fastapi import Body, Depends, Request
|
||||
import json
|
||||
|
||||
from fastapi import Body, Depends, Request, Response
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from ... import bundle, head, limits, models, schemas
|
||||
@@ -46,8 +48,38 @@ def export_adventure(
|
||||
|
||||
`app/bundle.py` owns the format, in all three of its versions. A backup
|
||||
outlives the schema, so no call site decides anything about its shape.
|
||||
|
||||
**v1.1 WP-D: the export also says whether this version could import it back.**
|
||||
A campaign large enough to pass `limits.MAX_IMPORT_BODY_BYTES` still exports —
|
||||
the file is complete and not damaged, and refusing to write it would destroy
|
||||
the only copy the reader was trying to make. What it cannot do is come back
|
||||
in here, and the reader is told that at the moment they take it rather than
|
||||
at the moment they need it.
|
||||
|
||||
It travels in headers, not in the body. The body is the bundle, the browser
|
||||
saves exactly those bytes as the file, and a warning inside it would become
|
||||
part of a portable story file and of every checksum taken over one.
|
||||
|
||||
The size measured is the compact serialisation, because that is both what
|
||||
this response sends and what the browser POSTs back on import, which is what
|
||||
`BodySizeLimitMiddleware` weighs. The pretty-printed file the reader
|
||||
downloads is larger, and is not what import reads.
|
||||
"""
|
||||
return bundle.export(db, adv)
|
||||
payload = bundle.export(db, adv)
|
||||
# Serialised exactly as Starlette's JSONResponse would, so the bytes counted
|
||||
# are the bytes sent.
|
||||
body = json.dumps(payload, ensure_ascii=False, allow_nan=False,
|
||||
separators=(",", ":")).encode("utf-8")
|
||||
limit = limits.MAX_IMPORT_BODY_BYTES
|
||||
importable = len(body) <= limit
|
||||
headers = {
|
||||
"X-Export-Bytes": str(len(body)),
|
||||
"X-Import-Limit-Bytes": str(limit),
|
||||
"X-Importable-By-This-Version": "true" if importable else "false",
|
||||
}
|
||||
if not importable:
|
||||
headers["X-Export-Warning"] = limits.oversized_export_warning(len(body), limit)
|
||||
return Response(content=body, media_type="application/json", headers=headers)
|
||||
|
||||
|
||||
@router.post("/import", response_model=schemas.ImportedAdventureOut, status_code=201)
|
||||
|
||||
@@ -0,0 +1,294 @@
|
||||
"""v1.1 WP-D: a backup that was really checked, and an export that says what it is.
|
||||
|
||||
Two recovery-path claims, each of which was true only in the small before this
|
||||
package:
|
||||
|
||||
- **A backup is verified.** M9 ran `PRAGMA quick_check` on the finished copy.
|
||||
That reads every page and every record, and skips the cross-check between a
|
||||
table and its indexes — so a copy whose index disagrees with its table passed.
|
||||
`test_the_fixture_is_the_difference_between_the_two_checks` builds exactly that
|
||||
damage and shows the two pragmas disagreeing about it, before anything here
|
||||
uses it as evidence.
|
||||
- **An export is importable.** Nothing compared the bundle with
|
||||
`limits.MAX_IMPORT_BODY_BYTES`, so a campaign could be exported and then
|
||||
refused by its own importer, with the reader finding out at the moment they
|
||||
needed it. The export still succeeds — the file is complete, and a version
|
||||
that refused to write it would destroy the copy someone was trying to make —
|
||||
and now it says so.
|
||||
|
||||
python -m pytest tests/test_v11_d_recovery.py -v
|
||||
"""
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from fastapi import Depends
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
from app import auth, backup, limits, models
|
||||
from app.database import Base, SessionLocal, engine, get_db
|
||||
from app.main import app
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def client():
|
||||
Base.metadata.create_all(bind=engine)
|
||||
setup = SessionLocal()
|
||||
user = models.User(is_guest=False, email="wp-d@example.com")
|
||||
setup.add(user)
|
||||
setup.flush()
|
||||
setup.add(models.Settings(user_id=user.id, model="test-model"))
|
||||
adventure = models.Adventure(user_id=user.id, title="Recovery")
|
||||
setup.add(adventure)
|
||||
setup.flush()
|
||||
setup.add(models.Action(adventure_id=adventure.id, type="start", text="The story opens."))
|
||||
setup.commit()
|
||||
adv_id, user_id = adventure.id, user.id
|
||||
setup.close()
|
||||
app.dependency_overrides[auth.get_current_user] = (
|
||||
lambda db=Depends(get_db): db.get(models.User, user_id))
|
||||
test_client = TestClient(app)
|
||||
test_client.adv_id = adv_id
|
||||
test_client.user_id = user_id
|
||||
try:
|
||||
yield test_client
|
||||
finally:
|
||||
app.dependency_overrides.clear()
|
||||
Base.metadata.drop_all(bind=engine)
|
||||
|
||||
|
||||
# ------------------------------------------------------------ the fixture
|
||||
|
||||
def build_corrupt_copy(path: Path) -> None:
|
||||
"""A database whose index disagrees with its table, and nothing else.
|
||||
|
||||
One digit inside one index leaf page is changed, so that entry names a key
|
||||
no row holds and one row's key is in no index entry. Every page is still
|
||||
structurally sound and every record still parses, which is the whole point:
|
||||
this is the damage `quick_check` is not looking for.
|
||||
"""
|
||||
path.unlink(missing_ok=True)
|
||||
connection = sqlite3.connect(path)
|
||||
connection.execute("PRAGMA page_size=4096")
|
||||
connection.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, k TEXT NOT NULL, filler TEXT)")
|
||||
connection.execute("CREATE INDEX i_t_k ON t(k)")
|
||||
connection.executemany("INSERT INTO t (k, filler) VALUES (?, ?)",
|
||||
[(f"k{n:06d}", "x" * 40) for n in range(400)])
|
||||
connection.commit()
|
||||
page_size = connection.execute("PRAGMA page_size").fetchone()[0]
|
||||
leaves = [row[0] for row in connection.execute(
|
||||
"SELECT pageno FROM dbstat WHERE name='i_t_k' AND pagetype='leaf' ORDER BY pageno")]
|
||||
connection.close()
|
||||
assert leaves, "the index must have a leaf page to damage"
|
||||
|
||||
raw = bytearray(path.read_bytes())
|
||||
start = (leaves[0] - 1) * page_size
|
||||
page = raw[start:start + page_size]
|
||||
at = page.find(b"k000")
|
||||
assert at != -1, "expected an indexed key on the index's first leaf page"
|
||||
page[at + 4] = ord("9") # k000144 -> k000944: a key no row has
|
||||
raw[start:start + page_size] = page
|
||||
path.write_bytes(bytes(raw))
|
||||
|
||||
|
||||
def check(path: Path, pragma: str) -> str:
|
||||
connection = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
|
||||
try:
|
||||
return ", ".join(str(row[0]) for row in connection.execute(f"PRAGMA {pragma}").fetchall())
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
|
||||
def test_the_fixture_is_the_difference_between_the_two_checks(tmp_path):
|
||||
"""Before using it as evidence: quick_check calls this database fine."""
|
||||
damaged = tmp_path / "damaged.db"
|
||||
build_corrupt_copy(damaged)
|
||||
assert check(damaged, "quick_check") == "ok"
|
||||
integrity = check(damaged, "integrity_check")
|
||||
assert integrity != "ok"
|
||||
assert "i_t_k" in integrity # it names the index that disagrees
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- backups
|
||||
|
||||
def test_a_healthy_backup_passes_the_full_check_and_is_kept(client, tmp_path):
|
||||
source = tmp_path / "campaign.db"
|
||||
source.write_bytes(Path(str(engine.url.database)).read_bytes())
|
||||
result = backup.create(source)
|
||||
assert result.integrity == "ok"
|
||||
assert result.path.exists() and result.bytes > 0
|
||||
assert check(result.path, "integrity_check") == "ok"
|
||||
assert result.path.parent == backup.directory(source)
|
||||
|
||||
|
||||
def test_the_backup_runs_the_full_check_not_the_quick_one(client, tmp_path, monkeypatch):
|
||||
"""The pragma itself, named. SQLite traces every statement it executes, so
|
||||
this reads what the backup actually asked the copy rather than inferring it."""
|
||||
asked: list[str] = []
|
||||
real_connect = sqlite3.connect
|
||||
|
||||
def tracing(*args, **kwargs):
|
||||
connection = real_connect(*args, **kwargs)
|
||||
connection.set_trace_callback(asked.append)
|
||||
return connection
|
||||
|
||||
monkeypatch.setattr(backup.sqlite3, "connect", tracing)
|
||||
source = tmp_path / "campaign.db"
|
||||
source.write_bytes(Path(str(engine.url.database)).read_bytes())
|
||||
backup.create(source).path.unlink()
|
||||
assert any("integrity_check" in sql for sql in asked), asked
|
||||
assert not any("quick_check" in sql for sql in asked), asked
|
||||
|
||||
|
||||
def test_a_copy_the_full_check_rejects_is_not_kept(client, tmp_path, monkeypatch):
|
||||
"""The copy is damaged after it is written and before it is verified, which
|
||||
is where a real page-level fault would appear: between the copy and the
|
||||
rename. Nothing wearing a backup's name may be left behind."""
|
||||
source = tmp_path / "campaign.db"
|
||||
source.write_bytes(Path(str(engine.url.database)).read_bytes())
|
||||
real_copy = backup._copy
|
||||
|
||||
def damage(source_path, working):
|
||||
pages = real_copy(source_path, working)
|
||||
build_corrupt_copy(working)
|
||||
return pages
|
||||
|
||||
monkeypatch.setattr(backup, "_copy", damage)
|
||||
with pytest.raises(backup.BackupError) as refused:
|
||||
backup.create(source)
|
||||
assert "did not verify" in str(refused.value)
|
||||
assert "i_t_k" in str(refused.value) # it says what was wrong
|
||||
kept = list(backup.directory(source).glob("*"))
|
||||
assert kept == [], f"a rejected backup was left behind: {kept}"
|
||||
|
||||
|
||||
def test_a_rejected_backup_leaves_an_earlier_good_one_alone(client, tmp_path, monkeypatch):
|
||||
source = tmp_path / "campaign.db"
|
||||
source.write_bytes(Path(str(engine.url.database)).read_bytes())
|
||||
good = backup.create(source)
|
||||
before = good.path.read_bytes()
|
||||
|
||||
real_copy = backup._copy
|
||||
|
||||
def damage(source_path, working):
|
||||
pages = real_copy(source_path, working)
|
||||
build_corrupt_copy(working)
|
||||
return pages
|
||||
|
||||
monkeypatch.setattr(backup, "_copy", damage)
|
||||
with pytest.raises(backup.BackupError):
|
||||
backup.create(source)
|
||||
assert good.path.exists()
|
||||
assert good.path.read_bytes() == before
|
||||
assert check(good.path, "integrity_check") == "ok"
|
||||
assert [p.name for p in backup.directory(source).glob("*")] == [good.path.name]
|
||||
|
||||
|
||||
def test_the_backup_file_semantics_are_unchanged(client, tmp_path):
|
||||
"""Same directory, same stamped name, same reported fields: WP-D changed the
|
||||
check, not the file."""
|
||||
source = tmp_path / "campaign.db"
|
||||
source.write_bytes(Path(str(engine.url.database)).read_bytes())
|
||||
first = backup.create(source)
|
||||
second = backup.create(source)
|
||||
assert first.path.name.startswith(backup.PREFIX) and first.path.suffix == ".db"
|
||||
assert first.path != second.path, "an existing backup is never overwritten"
|
||||
assert set(first.as_dict()) == {"filename", "bytes", "pages", "seconds", "integrity"}
|
||||
listed = [row["filename"] for row in backup.existing(source)]
|
||||
assert sorted(listed) == sorted([first.path.name, second.path.name])
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- exports
|
||||
|
||||
def export(client, adv_id):
|
||||
response = client.get(f"/api/adventures/{adv_id}/export")
|
||||
assert response.status_code == 200, response.text[:200]
|
||||
return response
|
||||
|
||||
|
||||
def test_a_normal_export_carries_no_warning(client):
|
||||
response = export(client, client.adv_id)
|
||||
assert "X-Export-Warning" not in response.headers
|
||||
assert response.headers["X-Importable-By-This-Version"] == "true"
|
||||
assert int(response.headers["X-Import-Limit-Bytes"]) == limits.MAX_IMPORT_BODY_BYTES
|
||||
assert int(response.headers["X-Export-Bytes"]) == len(response.content)
|
||||
assert response.json()["format"] == "ai-dnd-adventure-v3"
|
||||
|
||||
|
||||
def fill_past_the_limit(adv_id: int) -> int:
|
||||
"""Real rows, until the campaign's bundle is genuinely over the ceiling.
|
||||
|
||||
Not a mocked size: the export below serialises all of it.
|
||||
"""
|
||||
chunk = "The rain kept on over the harbour road, and nobody came. " * 900 # ~50 kB
|
||||
written = 0
|
||||
with SessionLocal() as db:
|
||||
while written < limits.MAX_IMPORT_BODY_BYTES + 2 * 1024 * 1024:
|
||||
db.add_all([models.Action(adventure_id=adv_id, type="ai", text=chunk)
|
||||
for _ in range(40)])
|
||||
db.commit()
|
||||
written += 40 * len(chunk)
|
||||
return written
|
||||
|
||||
|
||||
def test_an_oversized_export_is_still_delivered_and_says_it_cannot_come_back(client):
|
||||
fill_past_the_limit(client.adv_id)
|
||||
response = export(client, client.adv_id)
|
||||
|
||||
# Delivered, whole, and still the same format.
|
||||
body = response.content
|
||||
assert len(body) > limits.MAX_IMPORT_BODY_BYTES
|
||||
parsed = json.loads(body)
|
||||
assert parsed["format"] == "ai-dnd-adventure-v3"
|
||||
assert len(parsed["actions"]) > 40
|
||||
|
||||
# And honest about what this version can do with it.
|
||||
assert response.headers["X-Importable-By-This-Version"] == "false"
|
||||
warning = response.headers["X-Export-Warning"]
|
||||
assert limits.import_limit_label() in warning
|
||||
assert "exported successfully" in warning
|
||||
assert "cannot import" in warning
|
||||
assert int(response.headers["X-Export-Bytes"]) == len(body)
|
||||
|
||||
|
||||
def test_the_warning_follows_the_configured_limit(monkeypatch):
|
||||
"""The text is generated from the constant, so changing the constant changes
|
||||
the sentence rather than leaving a stale number in it."""
|
||||
assert "20 MB" in limits.oversized_export_warning(21_000_000)
|
||||
monkeypatch.setattr(limits, "MAX_IMPORT_BODY_BYTES", 50 * 1024 * 1024)
|
||||
assert limits.import_limit_label() == "50 MB"
|
||||
assert "50 MB" in limits.oversized_export_warning(60_000_000)
|
||||
assert "20 MB" not in limits.oversized_export_warning(60_000_000)
|
||||
|
||||
|
||||
def test_the_bundle_itself_never_carries_the_warning(client):
|
||||
"""The warning is about the export, not part of the portable story file."""
|
||||
fill_past_the_limit(client.adv_id)
|
||||
parsed = json.loads(export(client, client.adv_id).content)
|
||||
flat = json.dumps(parsed).lower()
|
||||
assert "import limit" not in flat
|
||||
assert "cannot import" not in flat
|
||||
for key in parsed:
|
||||
assert "warning" not in key.lower()
|
||||
|
||||
|
||||
def test_that_same_bundle_is_refused_by_import_naming_the_limit(client):
|
||||
fill_past_the_limit(client.adv_id)
|
||||
body = export(client, client.adv_id).content
|
||||
response = client.post("/api/adventures/import", content=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
assert response.status_code == 413
|
||||
detail = response.json()["detail"]
|
||||
assert "too large" in detail.lower()
|
||||
assert limits.import_limit_label() in detail
|
||||
|
||||
|
||||
def test_a_bundle_under_the_limit_still_imports(client):
|
||||
"""The refusal is about size alone: the ordinary path is untouched."""
|
||||
body = export(client, client.adv_id).content
|
||||
assert len(body) < limits.MAX_IMPORT_BODY_BYTES
|
||||
response = client.post("/api/adventures/import", content=body,
|
||||
headers={"Content-Type": "application/json"})
|
||||
assert response.status_code == 201, response.text[:200]
|
||||
@@ -0,0 +1,152 @@
|
||||
"""v1.1 WP-E: the contrast audit is a gate, not a report.
|
||||
|
||||
Before this package `tools/contrast_audit.py` measured control boundaries,
|
||||
printed that two of them were below 3:1, and exited 0 anyway — on the argument
|
||||
that a control is identified by its label rather than its edge. A check that
|
||||
cannot fail is not a check, and these tests are what make it one: the threshold
|
||||
is exercised from both sides, on a real tokens file, so a future palette change
|
||||
that dims a control's edge stops the run instead of adding a line to it.
|
||||
|
||||
python -m pytest tests/test_v11_e_contrast.py -v
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
|
||||
|
||||
import contrast_audit as audit # noqa: E402
|
||||
|
||||
|
||||
# --------------------------------------------------------------- the maths
|
||||
|
||||
def test_the_ratio_is_the_wcag_ratio():
|
||||
"""Anchored on values with known answers, so a broken formula is visible."""
|
||||
assert audit.ratio("#ffffff", "#000000") == pytest.approx(21.0, abs=0.01)
|
||||
assert audit.ratio("#ffffff", "#ffffff") == pytest.approx(1.0, abs=0.001)
|
||||
# Order does not matter: contrast is symmetric.
|
||||
assert audit.ratio("#131320", "#676792") == pytest.approx(
|
||||
audit.ratio("#676792", "#131320"), abs=1e-9)
|
||||
|
||||
|
||||
# ------------------------------------------------------- the gate itself
|
||||
|
||||
def tokens_file(tmp_path: Path, **overrides: str) -> Path:
|
||||
"""A real tokens file with named colours replaced."""
|
||||
source = audit.TOKENS.read_text()
|
||||
for name, value in overrides.items():
|
||||
token = "--" + name.replace("_", "-")
|
||||
start = source.index(f"{token}: ")
|
||||
end = source.index(";", start)
|
||||
source = source[:start] + f"{token}: {value}" + source[end:]
|
||||
written = tmp_path / "tokens.css"
|
||||
written.write_text(source)
|
||||
return written
|
||||
|
||||
|
||||
def run_against(path: Path, monkeypatch) -> int:
|
||||
monkeypatch.setattr(audit, "TOKENS", path)
|
||||
return audit.main()
|
||||
|
||||
|
||||
def test_a_boundary_below_three_to_one_fails_the_run(tmp_path, monkeypatch, capsys):
|
||||
"""2.99:1 against --bg-input — the wrong side of the line by one hundredth.
|
||||
|
||||
This value clears 3:1 against --bg-panel (3.21:1), so it would have passed
|
||||
the audit as M11 wrote it. It fails now because the floor is taken against
|
||||
the background the control is actually drawn on.
|
||||
"""
|
||||
below = tokens_file(tmp_path, border="#58639a")
|
||||
assert run_against(below, monkeypatch) == 1
|
||||
printed = capsys.readouterr().out
|
||||
assert "2.99:1" in printed
|
||||
assert "FAIL" in printed
|
||||
assert "below 3:1 (WCAG 1.4.11)" in printed
|
||||
|
||||
|
||||
def test_a_boundary_at_three_to_one_passes(tmp_path, monkeypatch, capsys):
|
||||
"""3.00:1 — the right side of the same line, one hundredth from the value
|
||||
above, and not failed for arithmetic the reader cannot see."""
|
||||
at = tokens_file(tmp_path, border="#58639b")
|
||||
assert run_against(at, monkeypatch) == 0
|
||||
printed = capsys.readouterr().out
|
||||
assert "3.00:1" in printed
|
||||
assert "every control boundary clears 3:1" in printed
|
||||
|
||||
|
||||
def test_the_v1_0_0_boundary_would_now_fail(tmp_path, monkeypatch, capsys):
|
||||
"""The value v1.0.0 shipped. This is the defect WP-E closes, and the gate
|
||||
has to be the thing that would have caught it."""
|
||||
shipped = tokens_file(tmp_path, border="#2b2b3d")
|
||||
assert run_against(shipped, monkeypatch) == 1
|
||||
assert "1.24:1" in capsys.readouterr().out # against --bg-input, the worst case
|
||||
|
||||
|
||||
def test_a_text_pair_below_four_point_five_still_fails(tmp_path, monkeypatch, capsys):
|
||||
"""WP-E raised the boundary floor and must not have lowered the text one."""
|
||||
dimmed = tokens_file(tmp_path, text_dim="#5a5750")
|
||||
assert run_against(dimmed, monkeypatch) == 1
|
||||
assert "below WCAG AA (1.4.3)" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_a_missing_token_is_a_failure_not_a_skip(tmp_path, monkeypatch, capsys):
|
||||
source = audit.TOKENS.read_text().replace("--border-bright:", "--border-was-renamed:")
|
||||
written = tmp_path / "tokens.css"
|
||||
written.write_text(source)
|
||||
assert run_against(written, monkeypatch) == 1
|
||||
assert "MISSING TOKEN" in capsys.readouterr().out
|
||||
|
||||
|
||||
# ------------------------------------------------- the shipped palette
|
||||
|
||||
def test_the_real_tokens_pass_both_criteria(capsys):
|
||||
"""The palette as it stands, through the same gate CI would run."""
|
||||
assert audit.main() == 0
|
||||
printed = capsys.readouterr().out
|
||||
assert "every text pair clears WCAG AA (1.4.3)" in printed
|
||||
assert "every control boundary clears 3:1 (1.4.11)" in printed
|
||||
assert "FAIL" not in printed
|
||||
|
||||
|
||||
def test_every_boundary_pair_is_measured_against_the_background_it_is_drawn_on():
|
||||
"""The audit used to check borders only against --bg-panel, which is not
|
||||
where the bordered controls are: inputs and buttons sit on --bg-input
|
||||
(styles/forms.css), which is lighter and therefore harder. Checking only the
|
||||
easier background would let a token pass while the real control failed."""
|
||||
boundary = [(fg, bg) for kind, fg, bg, _, _ in audit.PAIRS if kind == "boundary"]
|
||||
for background in ("--bg-input", "--bg-panel", "--bg"):
|
||||
assert ("--border", background) in boundary, background
|
||||
assert ("--border-bright", "--bg-input") in boundary
|
||||
|
||||
|
||||
def test_the_text_baselines_are_unchanged_by_wp_e():
|
||||
"""WP-E changed only boundary tokens. These are the M11 text measurements,
|
||||
and they have to still be exactly what the earlier reports recorded."""
|
||||
tokens = audit.read_tokens(audit.TOKENS)
|
||||
measured = {
|
||||
"body text on the page": audit.ratio(tokens["--text"], tokens["--bg"]),
|
||||
"body text in a panel": audit.ratio(tokens["--text"], tokens["--bg-panel"]),
|
||||
"secondary text in a panel": audit.ratio(tokens["--text-dim"], tokens["--bg-panel"]),
|
||||
"secondary text on the page": audit.ratio(tokens["--text-dim"], tokens["--bg"]),
|
||||
}
|
||||
assert measured["body text on the page"] == pytest.approx(14.57, abs=0.01)
|
||||
assert measured["body text in a panel"] == pytest.approx(13.57, abs=0.01)
|
||||
assert measured["secondary text in a panel"] == pytest.approx(5.48, abs=0.01)
|
||||
assert measured["secondary text on the page"] == pytest.approx(5.88, abs=0.01)
|
||||
|
||||
|
||||
def test_the_hover_edge_stays_brighter_than_the_resting_edge():
|
||||
"""Rest and hover have to remain distinguishable from each other, not merely
|
||||
each clear the floor against the background."""
|
||||
tokens = audit.read_tokens(audit.TOKENS)
|
||||
panel = tokens["--bg-panel"]
|
||||
assert audit.ratio(tokens["--border-bright"], panel) > audit.ratio(tokens["--border"], panel)
|
||||
|
||||
|
||||
def test_a_boundary_never_becomes_as_loud_as_body_text():
|
||||
"""A control's edge that outshines the words inside it is its own defect."""
|
||||
tokens = audit.read_tokens(audit.TOKENS)
|
||||
panel = tokens["--bg-panel"]
|
||||
assert audit.ratio(tokens["--border-bright"], panel) < audit.ratio(tokens["--text"], panel)
|
||||
@@ -26,16 +26,24 @@ TOKENS = Path(__file__).resolve().parent.parent.parent / "frontend/src/styles/to
|
||||
#: rather than combinatorial, because "every colour against every other" reports
|
||||
#: pairs that never meet on screen.
|
||||
#:
|
||||
#: The `kind` matters and is not a way of grading on a curve. **text** pairs are
|
||||
#: WCAG 1.4.3 Contrast (Minimum) and are what §21 of the M11 brief asks about;
|
||||
#: they are pass/fail. **boundary** pairs are WCAG 1.4.11 Non-text Contrast,
|
||||
#: which applies to "visual information required to identify user interface
|
||||
#: components" — and in this design a control is identified by its *label*,
|
||||
#: which is measured above and passes, not by its edge. So a boundary below 3:1
|
||||
#: is reported with its number and does not fail the run; what it would take to
|
||||
#: turn it into a real failure is a control with no visible label, and there is
|
||||
#: no such control (`tools/m11_browser.py` asserts every visible control has an
|
||||
#: accessible name, and the story controls are text buttons).
|
||||
#: The `kind` records which success criterion a pair is measured against —
|
||||
#: **text** is WCAG 1.4.3 Contrast (Minimum), **boundary** is WCAG 1.4.11
|
||||
#: Non-text Contrast — and **both are pass/fail**.
|
||||
#:
|
||||
#: v1.1 WP-E overturned the earlier position here, which was that a boundary
|
||||
#: below 3:1 could be recorded rather than failed because "a control is
|
||||
#: identified by its label, not by its edge". That argument understates what
|
||||
#: 1.4.11 asks: the criterion covers the visual information needed to identify
|
||||
#: a component *and its boundary*, and a reader who cannot see where a text box
|
||||
#: ends cannot see that there is a text box to type into, label or no label.
|
||||
#: The edges were at 1.33:1 and 1.75:1 — the tokens were raised instead.
|
||||
#:
|
||||
#: Borders are measured against **every background they are drawn on**, and the
|
||||
#: floor is the worst of them. Inputs and buttons sit on --bg-input, which is
|
||||
#: lighter than --bg-panel and so the harder case; checking only --bg-panel
|
||||
#: would have let a token pass the audit while the real control failed.
|
||||
#: `--bg-panel-glass` is translucent and cannot be resolved from tokens alone;
|
||||
#: that edge is measured on the rendered page by `tools/m11_browser.py`.
|
||||
PAIRS = [
|
||||
("text", "--text", "--bg", 4.5, "body text on the page"),
|
||||
("text", "--text", "--bg-panel", 4.5, "body text in a panel"),
|
||||
@@ -48,8 +56,14 @@ PAIRS = [
|
||||
("text", "--danger", "--bg-panel", 4.5, "an error message"),
|
||||
("text", "--warning", "--bg-panel", 4.5, "a caution message"),
|
||||
("text", "--player", "--bg", 4.5, "the player's own words"),
|
||||
("boundary", "--border", "--bg-panel", 3.0, "a control's resting edge"),
|
||||
("boundary", "--border-bright", "--bg-panel", 3.0, "a control's hover edge"),
|
||||
("boundary", "--border", "--bg-input", 3.0, "a field or button's resting edge"),
|
||||
("boundary", "--border-bright", "--bg-input", 3.0, "a field or button's hover edge"),
|
||||
("boundary", "--border", "--bg-panel", 3.0, "a control's resting edge in a panel"),
|
||||
("boundary", "--border-bright", "--bg-panel", 3.0, "a control's hover edge in a panel"),
|
||||
("boundary", "--border", "--bg", 3.0, "a divider on the page"),
|
||||
("boundary", "--border-bright", "--bg", 3.0, "the scrollbar thumb"),
|
||||
("boundary", "--accent-dim", "--bg-input", 3.0, "a focused field's edge"),
|
||||
("boundary", "--accent-dim", "--bg-panel", 3.0, "a focused control's edge in a panel"),
|
||||
("boundary", "--chart-1", "--bg-panel", 3.0, "a chart bar"),
|
||||
("boundary", "--chart-2", "--bg-panel", 3.0, "a chart bar"),
|
||||
("boundary", "--chart-3", "--bg-panel", 3.0, "a chart bar"),
|
||||
@@ -81,37 +95,44 @@ def ratio(a: str, b: str) -> float:
|
||||
|
||||
def main() -> int:
|
||||
tokens = read_tokens(TOKENS)
|
||||
print(f"{TOKENS.relative_to(TOKENS.parents[3])}: {len(tokens)} colour tokens\n")
|
||||
# Named defensively: the tests run this against a temporary tokens file,
|
||||
# which need not sit four directories deep the way the real one does.
|
||||
label = TOKENS.name
|
||||
if len(TOKENS.parents) > 3:
|
||||
label = TOKENS.relative_to(TOKENS.parents[3])
|
||||
print(f"{label}: {len(tokens)} colour tokens\n")
|
||||
print(f"{'pair':44} {'kind':9} {'ratio':>7} {'floor':>6} verdict")
|
||||
print("-" * 82)
|
||||
failures, advisories = 0, 0
|
||||
text_failures, boundary_failures = 0, 0
|
||||
for kind, foreground, background, floor, description in PAIRS:
|
||||
if foreground not in tokens or background not in tokens:
|
||||
print(f"{description:44} {kind:9} {'—':>7} {floor:>6.1f} MISSING TOKEN")
|
||||
failures += 1
|
||||
text_failures += 1
|
||||
continue
|
||||
measured = ratio(tokens[foreground], tokens[background])
|
||||
ok = measured >= floor
|
||||
if not ok:
|
||||
if kind == "text":
|
||||
failures += 1
|
||||
verdict = "FAIL"
|
||||
else:
|
||||
advisories += 1
|
||||
verdict = "below 1.4.11 (label carries it)"
|
||||
else:
|
||||
# Rounded to the two decimals printed, so the verdict matches what the
|
||||
# reader is shown: a pair reported as 3.00:1 is not failed for arithmetic
|
||||
# the output does not display.
|
||||
if round(measured, 2) >= floor:
|
||||
verdict = "pass"
|
||||
else:
|
||||
verdict = "FAIL"
|
||||
if kind == "text":
|
||||
text_failures += 1
|
||||
else:
|
||||
boundary_failures += 1
|
||||
print(f"{description:44} {kind:9} {measured:>6.2f}:1 {floor:>6.1f} {verdict}")
|
||||
print()
|
||||
if failures:
|
||||
print(f"{failures} text pair(s) below WCAG AA — this is a defect")
|
||||
if text_failures:
|
||||
print(f"{text_failures} text pair(s) below WCAG AA (1.4.3) — this is a defect")
|
||||
else:
|
||||
print("every text pair clears WCAG AA (1.4.3)")
|
||||
if advisories:
|
||||
print(f"{advisories} boundary pair(s) below 3:1 (1.4.11). Recorded rather "
|
||||
"than failed: every control in this design carries a visible text "
|
||||
"label, which is measured above and passes.")
|
||||
return 1 if failures else 0
|
||||
if boundary_failures:
|
||||
print(f"{boundary_failures} boundary pair(s) below 3:1 (WCAG 1.4.11) — "
|
||||
"this is a defect")
|
||||
else:
|
||||
print("every control boundary clears 3:1 (1.4.11)")
|
||||
return 1 if (text_failures or boundary_failures) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -665,6 +665,228 @@ def check_accessibility(browser: Browser, site: Site, adv: int, checks: Checks):
|
||||
checks.record("A11y", "the story input takes keyboard focus", typed is True)
|
||||
|
||||
|
||||
# -------------------------------------------------------------- WP-E boundaries
|
||||
|
||||
#: The controls whose edges WP-E measures, as the stylesheets actually draw them.
|
||||
#: `side` is the border the rule sets: .topnav paints only a bottom edge.
|
||||
BOUNDARY_TARGETS = [
|
||||
("E01", "the story composer", ".input-bar", "Top"),
|
||||
("E02", "a story control", ".story-controls button:not(:disabled)", "Top"),
|
||||
("E04", "the open panel tab", ".panel-tabs button.active", "Top"),
|
||||
]
|
||||
|
||||
#: `.topnav` is measured on the library route, because the play route does not
|
||||
#: render it — the play page has its own `.play-header`, which story.css keeps
|
||||
#: deliberately opaque. So the translucent case exists on exactly one screen,
|
||||
#: and it is the case token arithmetic cannot answer: --bg-panel-glass is
|
||||
#: rgba(...,0.82) over a gradient, so only the rendered page knows what is
|
||||
#: behind that edge.
|
||||
TRANSLUCENT_TARGET = ("E03", "the top navigation's edge", ".topnav", "Bottom")
|
||||
|
||||
#: Measured in the browser rather than from tokens, because two of these cannot
|
||||
#: be derived from tokens at all: .topnav sits on --bg-panel-glass, which is
|
||||
#: translucent, so its effective background is a composite of what is behind it;
|
||||
#: and a rendered edge can be changed by opacity, a transition mid-flight, or a
|
||||
#: rule the token file knows nothing about.
|
||||
#:
|
||||
#: A boundary is measured against **both** adjacent colours — the control's own
|
||||
#: fill inside it and the background outside it — and passes on the better of
|
||||
#: the two. An edge that matches its fill but contrasts with the page is still a
|
||||
#: visible outline, and vice versa; what 1.4.11 asks is that the component's
|
||||
#: extent be perceivable, not that every neighbouring surface differ from it.
|
||||
BOUNDARY_JS = """
|
||||
const parse = (c) => (c.match(/[\\d.]+/g) || []).map(Number);
|
||||
const alphaOf = (c) => {
|
||||
if (!c || c === 'transparent') return 0;
|
||||
const p = parse(c);
|
||||
return p.length > 3 ? p[3] : 1;
|
||||
};
|
||||
function lum(c) {
|
||||
const [r, g, b] = parse(c).slice(0, 3).map(v => v / 255)
|
||||
.map(v => v <= 0.03928 ? v / 12.92 : Math.pow((v + 0.055) / 1.055, 2.4));
|
||||
return 0.2126 * r + 0.7152 * g + 0.0722 * b;
|
||||
}
|
||||
function over(fg, bg) {
|
||||
const f = parse(fg), b = parse(bg), a = alphaOf(fg);
|
||||
return 'rgb(' + [0, 1, 2].map(i => Math.round(f[i] * a + b[i] * (1 - a))).join(', ') + ')';
|
||||
}
|
||||
function ratio(x, y) {
|
||||
const a = lum(x), b = lum(y);
|
||||
return (Math.max(a, b) + 0.05) / (Math.min(a, b) + 0.05);
|
||||
}
|
||||
// Everything painted behind `el`, composited bottom-up, so a translucent
|
||||
// panel reports the colour a reader actually sees rather than its own rgba.
|
||||
function behind(el) {
|
||||
const layers = [];
|
||||
let node = el;
|
||||
while (node && node !== document.documentElement) {
|
||||
const c = getComputedStyle(node).backgroundColor;
|
||||
if (alphaOf(c) > 0) {
|
||||
layers.push(c);
|
||||
if (alphaOf(c) >= 1) break;
|
||||
}
|
||||
node = node.parentElement;
|
||||
}
|
||||
let result = 'rgb(10, 10, 15)';
|
||||
const root = getComputedStyle(document.documentElement).backgroundColor;
|
||||
const body = getComputedStyle(document.body).backgroundColor;
|
||||
if (alphaOf(root) >= 1) result = root;
|
||||
else if (alphaOf(body) >= 1) result = body;
|
||||
for (let i = layers.length - 1; i >= 0; i--) {
|
||||
result = alphaOf(layers[i]) >= 1 ? layers[i] : over(layers[i], result);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
const [selector, side] = arguments;
|
||||
const el = document.querySelector(selector);
|
||||
if (!el) return null;
|
||||
const s = getComputedStyle(el);
|
||||
const edge = s['border' + side + 'Color'];
|
||||
const width = parseFloat(s['border' + side + 'Width']) || 0;
|
||||
const opacity = parseFloat(s.opacity);
|
||||
const inside = alphaOf(s.backgroundColor) >= 1
|
||||
? s.backgroundColor : over(s.backgroundColor, behind(el.parentElement || el));
|
||||
const outside = behind(el.parentElement || el);
|
||||
return {
|
||||
selector, edge, width, opacity, inside, outside,
|
||||
insideRatio: Math.round(ratio(edge, inside) * 100) / 100,
|
||||
outsideRatio: Math.round(ratio(edge, outside) * 100) / 100,
|
||||
outline: s.outlineStyle + ' ' + s.outlineWidth + ' ' + s.outlineColor,
|
||||
shadow: s.boxShadow,
|
||||
};
|
||||
"""
|
||||
|
||||
|
||||
def _settled(browser: Browser, selector: str, side: str, *, timeout: float = 5) -> None:
|
||||
"""Waits until the edge colour stops moving.
|
||||
|
||||
Every one of these controls carries `transition: border-color 0.15s`, so a
|
||||
measurement taken the instant after a hover or a focus reads a colour part
|
||||
way between the two states — a value no state actually has. The first run of
|
||||
this check did exactly that: it reported the hover edge as rgb(114,114,160),
|
||||
which is neither --border nor --border-bright but a frame between them.
|
||||
Polled rather than slept, per this harness's rule.
|
||||
"""
|
||||
script = ("(() => { const el = document.querySelector(%s);"
|
||||
" if (!el) return true;"
|
||||
" const c = getComputedStyle(el)['border%sColor'];"
|
||||
" const was = window.__wpeEdge; window.__wpeEdge = c;"
|
||||
" return was === c; })()" % (json.dumps(selector), side))
|
||||
# Cleared first, so the comparison starts from no previous reading rather
|
||||
# than from whatever the last measured control left behind.
|
||||
browser.js("window.__wpeEdge = undefined")
|
||||
browser.wait_js(script, timeout=timeout)
|
||||
|
||||
|
||||
def _boundary(browser: Browser, selector: str, side: str) -> dict | None:
|
||||
_settled(browser, selector, side)
|
||||
return browser.js(BOUNDARY_JS, selector, side)
|
||||
|
||||
|
||||
def _record_boundary(checks: Checks, test: str, what: str, row: dict | None) -> None:
|
||||
if row is None:
|
||||
checks.skip(test, what, "the control was not on the page")
|
||||
return
|
||||
best = max(row["insideRatio"], row["outsideRatio"])
|
||||
detail = (f"{best:.2f}:1 (edge {row['edge']} — {row['insideRatio']}:1 against its fill "
|
||||
f"{row['inside']}, {row['outsideRatio']}:1 against {row['outside']}), "
|
||||
f"{row['width']}px")
|
||||
checks.record("WCAG 1.4.11", what, best >= 3.0 and row["width"] > 0, detail)
|
||||
|
||||
|
||||
def check_control_boundaries(browser: Browser, site: Site, adv: int, checks: Checks,
|
||||
evidence: dict, out: Path) -> None:
|
||||
"""v1.1 WP-E §21: a control's edge is visible on its own, measured.
|
||||
|
||||
M11 measured text contrast on the rendered page and left boundaries to the
|
||||
token audit, which checked them against one background and reported a
|
||||
shortfall without failing. These are the rendered edges, at rest, on hover
|
||||
and while focused, against what is actually behind them.
|
||||
"""
|
||||
_open_play(browser, site, adv)
|
||||
measured: dict[str, dict] = {}
|
||||
|
||||
# A panel tab's edge is `transparent` until the panel is open, which makes
|
||||
# the open tab the one control here whose boundary is the only thing marking
|
||||
# it — exactly what 1.4.11 is about. So one is opened rather than skipped.
|
||||
if not _open_panel(browser, "State"):
|
||||
checks.skip("E04", "the open panel tab — resting edge", "the State panel did not open")
|
||||
|
||||
for test, what, selector, side in BOUNDARY_TARGETS:
|
||||
row = _boundary(browser, selector, side)
|
||||
_record_boundary(checks, test, f"{what} — resting edge", row)
|
||||
if row:
|
||||
measured[f"{what} (rest)"] = row
|
||||
|
||||
# Hover, with a real pointer: `:hover` follows the browser's pointer state,
|
||||
# so a synthetic mouseover would silently re-measure the resting edge.
|
||||
control = browser.find(".story-controls button:not(:disabled)", required=False)
|
||||
if control is None:
|
||||
checks.skip("E02", "a story control — hover edge", "no enabled control on the page")
|
||||
else:
|
||||
browser.hover(control)
|
||||
hovered = browser.js(
|
||||
"return document.querySelector('.story-controls button:not(:disabled)')"
|
||||
" .matches(':hover');")
|
||||
if hovered is not True:
|
||||
checks.skip("E02", "a story control — hover edge",
|
||||
"the pointer did not land on the control")
|
||||
else:
|
||||
row = _boundary(browser, ".story-controls button:not(:disabled)", "Top")
|
||||
_record_boundary(checks, "E02", "a story control — hover edge", row)
|
||||
if row:
|
||||
measured["a story control (hover)"] = row
|
||||
browser.unhover()
|
||||
|
||||
# Focus: the composer's edge changes colour while it holds focus, and that
|
||||
# edge is what tells a keyboard reader where they are.
|
||||
focused = browser.js("""
|
||||
const box = document.querySelector('.input-main textarea');
|
||||
if (!box) return false;
|
||||
box.focus();
|
||||
return document.activeElement === box;
|
||||
""")
|
||||
if focused is not True:
|
||||
checks.skip("E01", "the story composer — focused edge", "the composer did not take focus")
|
||||
else:
|
||||
row = _boundary(browser, ".input-bar", "Top")
|
||||
_record_boundary(checks, "E01", "the story composer — focused edge", row)
|
||||
if row:
|
||||
measured["the story composer (focus)"] = row
|
||||
# A focus ring that is only a colour change is not enough on its own;
|
||||
# M11 already asserts a visible focus indicator, and this says the
|
||||
# focused edge is also measurably distinct from the resting one.
|
||||
rest = measured.get("the story composer (rest)")
|
||||
if rest:
|
||||
checks.record("WCAG 1.4.11", "the focused composer edge differs from its resting edge",
|
||||
row["edge"] != rest["edge"],
|
||||
f"rest {rest['edge']} -> focus {row['edge']}")
|
||||
|
||||
shot = browser.screenshot(out / "control-boundaries.png")
|
||||
checks.record("WP-E", "a screenshot of the measured controls was captured",
|
||||
shot.exists() and shot.stat().st_size > 0, str(shot))
|
||||
|
||||
# The translucent edge, on the only screen that has it. Token arithmetic
|
||||
# cannot reach this one: --bg-panel-glass is rgba over a gradient, so what
|
||||
# is behind the nav's bottom edge is known only to the rendered page.
|
||||
test, what, selector, side = TRANSLUCENT_TARGET
|
||||
browser.go(site.url)
|
||||
browser.wait_for(".topnav", timeout=30)
|
||||
row = _boundary(browser, selector, side)
|
||||
_record_boundary(checks, test, f"{what} — over a translucent panel", row)
|
||||
if row:
|
||||
measured[f"{what} (translucent)"] = row
|
||||
checks.record("WP-E", "the nav's background really is translucent",
|
||||
row["outside"] != row["inside"],
|
||||
f"composited to {row['inside']} over {row['outside']}")
|
||||
nav_shot = browser.screenshot(out / "control-boundaries-nav.png")
|
||||
checks.record("WP-E", "a screenshot of the navigation edge was captured",
|
||||
nav_shot.exists() and nav_shot.stat().st_size > 0, str(nav_shot))
|
||||
|
||||
evidence["control_boundaries"] = {"measured": measured,
|
||||
"screenshots": [str(shot), str(nav_shot)]}
|
||||
|
||||
|
||||
# -------------------------------------------------------------- WP-C scenarios
|
||||
|
||||
def _take_count_is(count: str) -> str:
|
||||
@@ -1354,7 +1576,11 @@ def main() -> int:
|
||||
"C5", "failed generation and recovery"),
|
||||
("export", lambda: check_export_download(browser, site, adv, checks, evidence, out)),
|
||||
]
|
||||
for suite, scenarios in (("M11", m11), ("WP-C", wpc)):
|
||||
wpe = [
|
||||
("boundaries", lambda: check_control_boundaries(browser, site, adv, checks,
|
||||
evidence, out)),
|
||||
]
|
||||
for suite, scenarios in (("M11", m11), ("WP-C", wpc), ("WP-E", wpe)):
|
||||
checks.suite = suite
|
||||
for name, scenario in scenarios:
|
||||
if not wanted(name):
|
||||
@@ -1383,7 +1609,8 @@ def main() -> int:
|
||||
"kind": ("release regression" if narrated and not only else
|
||||
"partial (no narrator)" if not narrated else f"development (only {sorted(only)})"),
|
||||
"checks": checks.rows,
|
||||
"suites": {"M11": checks.counts("M11"), "WP-C": checks.counts("WP-C")},
|
||||
"suites": {"M11": checks.counts("M11"), "WP-C": checks.counts("WP-C"),
|
||||
"WP-E": checks.counts("WP-E")},
|
||||
"passed": checks.counts()["passed"],
|
||||
"failed": checks.counts()["failed"],
|
||||
"skipped": checks.counts()["skipped"],
|
||||
|
||||
@@ -25,6 +25,7 @@ has actually finished arriving — never the click that started it.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
@@ -226,6 +227,45 @@ class Browser:
|
||||
def source(self) -> str:
|
||||
return self._call("GET", self._s("/source"))["value"]
|
||||
|
||||
def screenshot(self, path) -> Path:
|
||||
"""The viewport as a PNG, written where you ask (v1.1 WP-E).
|
||||
|
||||
Evidence for a change a reader judges by looking at it: a contrast ratio
|
||||
says a boundary is measurable, and a picture says what it looks like.
|
||||
"""
|
||||
encoded = self._call("GET", self._s("/screenshot"))["value"]
|
||||
target = Path(path)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes(base64.b64decode(encoded))
|
||||
return target
|
||||
|
||||
def hover(self, element: str) -> None:
|
||||
"""A real pointer over an element, so `:hover` actually applies.
|
||||
|
||||
Dispatching a mouseover event from JavaScript does not do this: CSS
|
||||
`:hover` follows the browser's own pointer state, not a synthetic event,
|
||||
so a measurement taken after `dispatchEvent` reads the resting style and
|
||||
reports it as the hover style. This moves the pointer (v1.1 WP-E).
|
||||
"""
|
||||
self._call("POST", self._s("/execute/sync"), {
|
||||
"script": "arguments[0].scrollIntoView({block: 'center', inline: 'nearest'})",
|
||||
"args": [{ELEMENT_KEY: element}]})
|
||||
self._call("POST", self._s("/actions"), {"actions": [{
|
||||
"type": "pointer", "id": "mouse", "parameters": {"pointerType": "mouse"},
|
||||
"actions": [{"type": "pointerMove", "duration": 60,
|
||||
"origin": {ELEMENT_KEY: element}, "x": 0, "y": 0}]}]})
|
||||
|
||||
def unhover(self) -> None:
|
||||
"""Move the pointer off whatever it was over, and forget the input state."""
|
||||
self._call("POST", self._s("/actions"), {"actions": [{
|
||||
"type": "pointer", "id": "mouse", "parameters": {"pointerType": "mouse"},
|
||||
"actions": [{"type": "pointerMove", "duration": 30,
|
||||
"origin": "viewport", "x": 0, "y": 0}]}]})
|
||||
try:
|
||||
self._call("DELETE", self._s("/actions"))
|
||||
except WebDriverError:
|
||||
pass
|
||||
|
||||
def js(self, script: str, *args):
|
||||
return self._call("POST", self._s("/execute/sync"),
|
||||
{"script": script, "args": list(args)})["value"]
|
||||
|
||||
@@ -0,0 +1,353 @@
|
||||
"""v1.1 WP-D criterion 3: the backup completes **through the UI**.
|
||||
|
||||
python -m tools.wpd_backup_ui --out <dir under $HOME> [--case real|large|both] [--show]
|
||||
|
||||
Run from `backend/`, with `frontend/dist` already built.
|
||||
|
||||
WP-D's first pass drove `POST /api/backups` — the endpoint the *Back up now*
|
||||
button calls — on a 2.3 MB campaign database and a 117 MB one. That is evidence
|
||||
about the implementation, and the plan's criterion 3 asks for something else:
|
||||
that the backup *completes through the UI* on both. A reader does not call an
|
||||
endpoint. This drives the reader-facing control in a real Firefox, against the
|
||||
production build served by FastAPI, exactly as WP-C's harness does.
|
||||
|
||||
**No narrator and no inference.** A backup needs neither, so nothing here
|
||||
touches a model host.
|
||||
|
||||
What it refuses to call a pass, per the brief:
|
||||
|
||||
- the click does nothing (no toast, no file);
|
||||
- the request fails (an error toast);
|
||||
- no backup file appears on disk;
|
||||
- the finished copy does not pass a full `PRAGMA integrity_check`;
|
||||
- the UI reports an error.
|
||||
|
||||
Every wait is on a condition the page or the filesystem can show. Nothing here
|
||||
sleeps and then asserts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
||||
|
||||
from tools.m11_webdriver import ( # noqa: E402
|
||||
Browser, Site, geckodriver_version, require_under_home,
|
||||
)
|
||||
|
||||
BACKEND = Path(__file__).resolve().parent.parent
|
||||
HOME = Path.home()
|
||||
|
||||
#: The two databases criterion 3 names. Both are copied before use: the first is
|
||||
#: the M11 evidence campaign and must not be written to, and the second is the
|
||||
#: 117 MB application database built for WP-D's timing.
|
||||
DEFAULT_REAL = HOME / "m11-evidence/m04-final/campaign.db"
|
||||
DEFAULT_LARGE = HOME / "v11-evidence/wp-d/endpoint/campaign-100mb/campaign.db"
|
||||
|
||||
BLOCK = '[data-testid="database-backup"]'
|
||||
BUTTON = f'{BLOCK} button.primary'
|
||||
|
||||
|
||||
class Checks:
|
||||
"""Results, with the discipline that an unrun check is not a passing one."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.rows: list[dict] = []
|
||||
|
||||
def record(self, case: str, name: str, ok: bool, detail: str = "") -> bool:
|
||||
self.rows.append({"case": case, "check": name,
|
||||
"result": "PASS" if ok else "FAIL", "detail": detail})
|
||||
print(f" {'ok ' if ok else 'FAIL'} {case:6} {name}"
|
||||
+ (f" — {detail}" if detail else ""), flush=True)
|
||||
return ok
|
||||
|
||||
@property
|
||||
def failed(self) -> list[dict]:
|
||||
return [r for r in self.rows if r["result"] == "FAIL"]
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- helpers
|
||||
|
||||
def integrity_of(path: Path) -> tuple[str, float]:
|
||||
"""The full check on a finished copy, and what it cost, measured here.
|
||||
|
||||
The application runs its own `integrity_check` before keeping the file; this
|
||||
is an independent second opinion on the artefact the UI produced, and it is
|
||||
where criterion 3's 'time for integrity_check' comes from.
|
||||
"""
|
||||
connection = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
|
||||
try:
|
||||
started = time.perf_counter()
|
||||
rows = connection.execute("PRAGMA integrity_check").fetchall()
|
||||
elapsed = time.perf_counter() - started
|
||||
finally:
|
||||
connection.close()
|
||||
return ", ".join(str(r[0]) for r in rows), elapsed
|
||||
|
||||
|
||||
def digest(path: Path) -> str:
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()[:16]
|
||||
|
||||
|
||||
def backups_in(db_path: Path) -> dict[str, int]:
|
||||
directory = db_path.parent / "backups"
|
||||
if not directory.exists():
|
||||
return {}
|
||||
return {p.name: p.stat().st_size for p in directory.iterdir() if p.is_file()}
|
||||
|
||||
|
||||
def wait_for_new_backup(db_path: Path, before: set[str], *, timeout: float = 300,
|
||||
poll: float = 0.2) -> Path | None:
|
||||
"""The backup file the application wrote, once it is really there.
|
||||
|
||||
A file condition, not a sleep: a name that was not there before, not a
|
||||
`.partial`, and a size that has stopped growing.
|
||||
"""
|
||||
directory = db_path.parent / "backups"
|
||||
deadline = time.monotonic() + timeout
|
||||
sizes: dict[str, int] = {}
|
||||
while time.monotonic() < deadline:
|
||||
if directory.exists():
|
||||
for path in directory.iterdir():
|
||||
if not path.is_file() or path.name in before:
|
||||
continue
|
||||
if path.name.endswith(".partial"):
|
||||
continue
|
||||
size = path.stat().st_size
|
||||
if size > 0 and sizes.get(path.name) == size:
|
||||
return path
|
||||
sizes[path.name] = size
|
||||
time.sleep(poll)
|
||||
return None
|
||||
|
||||
|
||||
def toasts(browser: Browser) -> list[dict]:
|
||||
"""What the page is telling the reader — the message apart from its mark.
|
||||
|
||||
A toast renders a decorative mark before the message:
|
||||
`<span class="toast-mark" aria-hidden="true">❖</span><span>…</span>`. So the
|
||||
button's `textContent` begins with that character, and the first run of this
|
||||
tool matched `textContent.startswith('Backup written:')` and reported six
|
||||
*passing* behaviours as failures — the UI had said exactly what it should,
|
||||
and the assertion was reading the mark. `message` is the message span alone;
|
||||
`text` is kept whole for the evidence record.
|
||||
"""
|
||||
return browser.js("""
|
||||
const host = document.querySelector('.toast-host');
|
||||
if (!host) return [];
|
||||
return [...host.querySelectorAll('button.toast')].map(b => {
|
||||
const span = b.querySelector('span:not(.toast-mark)');
|
||||
return {
|
||||
error: b.classList.contains('toast-error'),
|
||||
message: (span ? span.textContent : b.textContent).trim(),
|
||||
text: b.textContent.trim(),
|
||||
};
|
||||
});
|
||||
""") or []
|
||||
|
||||
|
||||
def open_backup_panel(browser: Browser, site: Site, checks: Checks, case: str) -> bool:
|
||||
"""Reach the control the way a reader does: the nav link, then the panel."""
|
||||
browser.go(site.url)
|
||||
browser.wait_for(".topnav", timeout=60)
|
||||
link = browser.find('.nav-links a[href="/settings"]', required=False)
|
||||
if link is None:
|
||||
return checks.record(case, "the Settings link is in the navigation", False)
|
||||
browser.click(link)
|
||||
opened = browser.wait_js(f"!!document.querySelector('{BLOCK}')", timeout=60)
|
||||
if not checks.record(case, "Settings opens from the navigation link", opened,
|
||||
browser.url):
|
||||
return False
|
||||
|
||||
summary = browser.find(f"{BLOCK} summary", required=False)
|
||||
if summary is None:
|
||||
return checks.record(case, "the backup panel has a disclosure", False)
|
||||
browser.click(summary)
|
||||
# The panel is a <details>: it loads what is already on disk when it opens,
|
||||
# so waiting for the directory line proves the application answered.
|
||||
shown = browser.wait_js(
|
||||
f"document.querySelector('{BLOCK}').open === true"
|
||||
f" && !!document.querySelector('{BUTTON}')", timeout=30)
|
||||
checks.record(case, "the backup panel opens and shows its control", shown)
|
||||
listed = browser.wait_js(f"!!document.querySelector('{BLOCK} code')", timeout=30)
|
||||
checks.record(case, "the panel reports where backups are written", listed)
|
||||
return shown
|
||||
|
||||
|
||||
def click_back_up_now(browser: Browser, site: Site, db: Path, checks: Checks,
|
||||
case: str, label: str) -> dict:
|
||||
"""One press of the button, judged by what the page and the disk then show."""
|
||||
before = set(backups_in(db))
|
||||
# Clear anything still on screen, so the toast this press produces is the
|
||||
# one that is read back rather than a leftover from the previous press.
|
||||
browser.js("""
|
||||
for (const b of document.querySelectorAll('.toast-host button.toast')) b.click();
|
||||
return true;
|
||||
""")
|
||||
button = browser.find(BUTTON, required=False)
|
||||
if button is None:
|
||||
checks.record(case, f"{label}: the control is on the page", False)
|
||||
return {}
|
||||
|
||||
started = time.perf_counter()
|
||||
browser.click(button)
|
||||
|
||||
# Either outcome ends the wait, so a failure is reported as a failure rather
|
||||
# than as a timeout.
|
||||
settled = browser.wait_js(
|
||||
"(() => { const t = [...document.querySelectorAll('.toast-host button.toast')];"
|
||||
" return t.length > 0; })()", timeout=300)
|
||||
elapsed = time.perf_counter() - started
|
||||
|
||||
shown = toasts(browser)
|
||||
errors = [t for t in shown if t["error"]]
|
||||
written = [t for t in shown if t["message"].startswith("Backup written:")]
|
||||
|
||||
checks.record(case, f"{label}: the click produced a visible result", settled,
|
||||
json.dumps(shown)[:200])
|
||||
checks.record(case, f"{label}: the UI reports no error",
|
||||
not errors, json.dumps(errors)[:300])
|
||||
checks.record(case, f"{label}: the UI reports the backup was written",
|
||||
bool(written), json.dumps(written)[:200])
|
||||
|
||||
idle = browser.wait_js(
|
||||
"(() => { const b = document.querySelector(%s);"
|
||||
" return !!b && b.textContent.trim() === 'Back up now'; })()"
|
||||
% json.dumps(BUTTON), timeout=120)
|
||||
checks.record(case, f"{label}: the control returns from 'Backing up…'", idle)
|
||||
|
||||
produced = wait_for_new_backup(db, before)
|
||||
checks.record(case, f"{label}: a backup file was physically produced",
|
||||
produced is not None, str(produced))
|
||||
if produced is None:
|
||||
return {"seconds": round(elapsed, 3), "toasts": shown}
|
||||
|
||||
named = any(produced.name in t["message"] for t in written)
|
||||
checks.record(case, f"{label}: the UI names the file that appeared", named,
|
||||
f"{produced.name} — {written[0]['message'] if written else ''}")
|
||||
|
||||
relisted = browser.wait_js(
|
||||
"[...document.querySelectorAll('%s .backup-list code')]"
|
||||
".some(c => c.textContent.trim() === %s)" % (BLOCK, json.dumps(produced.name)),
|
||||
timeout=60)
|
||||
checks.record(case, f"{label}: the new backup appears in the panel's list", relisted)
|
||||
|
||||
verdict, integrity_seconds = integrity_of(produced)
|
||||
checks.record(case, f"{label}: the finished copy passes full integrity_check",
|
||||
verdict == "ok", f"{verdict} in {integrity_seconds * 1000:.1f} ms")
|
||||
|
||||
return {
|
||||
"file": str(produced),
|
||||
"bytes": produced.stat().st_size,
|
||||
"sha256_16": digest(produced),
|
||||
"seconds": round(elapsed, 3),
|
||||
"integrity": verdict,
|
||||
"integrity_seconds": round(integrity_seconds, 4),
|
||||
"toasts": shown,
|
||||
}
|
||||
|
||||
|
||||
def run_case(case: str, source: Path, out: Path, checks: Checks, *, show: bool,
|
||||
twice: bool) -> dict:
|
||||
print(f"\n=== {case}: {source} ===", flush=True)
|
||||
work = out / case
|
||||
shutil.rmtree(work, ignore_errors=True)
|
||||
(work / "data").mkdir(parents=True)
|
||||
db = work / "data" / "campaign.db"
|
||||
copy_started = time.perf_counter()
|
||||
shutil.copy2(source, db)
|
||||
copy_seconds = time.perf_counter() - copy_started
|
||||
size = db.stat().st_size
|
||||
print(f" copied {size:,} bytes in {copy_seconds:.2f}s -> {db}", flush=True)
|
||||
|
||||
site = Site(BACKEND, db, work / "server.log")
|
||||
browser = Browser(headless=not show, log=work / "geckodriver.log")
|
||||
result: dict = {"database": str(source), "bytes": size,
|
||||
"served_at": site.url, "firefox": browser.version}
|
||||
try:
|
||||
if not open_backup_panel(browser, site, checks, case):
|
||||
return result
|
||||
result["first"] = click_back_up_now(browser, site, db, checks, case, "backup")
|
||||
browser.screenshot(work / "back-up-now.png")
|
||||
|
||||
if twice and result["first"].get("file"):
|
||||
kept = Path(result["first"]["file"])
|
||||
before_bytes, before_digest = kept.stat().st_size, digest(kept)
|
||||
result["second"] = click_back_up_now(browser, site, db, checks, case,
|
||||
"second backup")
|
||||
still_there = kept.exists()
|
||||
checks.record(case, "the earlier backup still exists", still_there)
|
||||
if still_there:
|
||||
checks.record(
|
||||
case, "and is byte-for-byte what it was",
|
||||
kept.stat().st_size == before_bytes and digest(kept) == before_digest,
|
||||
f"{before_bytes:,} bytes, sha256:{before_digest}")
|
||||
verdict, _ = integrity_of(kept)
|
||||
checks.record(case, "and still passes integrity_check", verdict == "ok",
|
||||
verdict)
|
||||
if result["second"].get("file"):
|
||||
checks.record(case, "the second backup is a different file",
|
||||
result["second"]["file"] != result["first"]["file"],
|
||||
Path(result["second"]["file"]).name)
|
||||
finally:
|
||||
browser.quit()
|
||||
site.stop()
|
||||
return result
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--out", required=True,
|
||||
help="evidence directory, which must be under $HOME")
|
||||
parser.add_argument("--case", default="both", choices=["real", "large", "both"])
|
||||
parser.add_argument("--show", action="store_true", help="run Firefox visibly")
|
||||
parser.add_argument("--real-db", default=str(DEFAULT_REAL))
|
||||
parser.add_argument("--large-db", default=str(DEFAULT_LARGE))
|
||||
args = parser.parse_args()
|
||||
|
||||
out = require_under_home(Path(args.out).expanduser())
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
if not (BACKEND.parent / "frontend/dist/index.html").exists():
|
||||
print("frontend/dist is not built", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
checks = Checks()
|
||||
started = datetime.now()
|
||||
results: dict[str, dict] = {}
|
||||
wanted = [("real", Path(args.real_db).expanduser(), True)] if args.case != "large" else []
|
||||
if args.case != "real":
|
||||
wanted.append(("large", Path(args.large_db).expanduser(), False))
|
||||
|
||||
print(f"WP-D criterion 3 — the backup through the UI. geckodriver "
|
||||
f"{geckodriver_version()}", flush=True)
|
||||
for case, source, twice in wanted:
|
||||
if not source.exists():
|
||||
checks.record(case, "the database is present", False, str(source))
|
||||
continue
|
||||
results[case] = run_case(case, source, out, checks, show=args.show, twice=twice)
|
||||
|
||||
report = {
|
||||
"started": started.isoformat(timespec="seconds"),
|
||||
"seconds": round((datetime.now() - started).total_seconds()),
|
||||
"cases": results,
|
||||
"checks": checks.rows,
|
||||
"passed": len([r for r in checks.rows if r["result"] == "PASS"]),
|
||||
"failed": len(checks.failed),
|
||||
}
|
||||
(out / "backup-ui-report.json").write_text(json.dumps(report, indent=2))
|
||||
print(f"\n{report['passed']} passed, {report['failed']} failed "
|
||||
f"-> {out / 'backup-ui-report.json'}")
|
||||
return 1 if checks.failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user