Stop a turn locking out its own memory bank, and let the long run notice

The first M01 trial with the memory bank on was 26 turns on a GPU host. It
accepted every turn and reported "complete". It also wrote two memories and
no summary, and logged 180 `database is locked` errors, while derived status
still read `idle`.

The cause was a single uncommitted UPDATE. Retrieval bumped each used
memory's counter before the model call, and the turn commits only after the
reply has streamed. SQLite has one writer, so the turn held the write lock for
the whole reply. Every post-turn memory, summary and status write in that
window waited out the five-second timeout and failed. Recording the failure
needed a write as well, and without a rollback first it raised
PendingRollbackError. The loss therefore reached the log and never reached
the status the Insights panel reads, which F08 forbids. The draco run never
hit this because the bank was off there.

- `retrieve_memories` now only reads. `record_use` writes the counters in the
  turn's single commit, so a turn that never lands counts nothing.
- The post-turn task's outer handler rolls back before it records a failure.

The harness could not have caught any of this. It read three prompt sections
under names the builder does not use: `memories` (really `used_memories`),
`story_history` (really `history`/`recent_history`), and a `knowledge` prefix
that matched the fixed instruction section instead of the imported passages.
Memory tokens read 0 whatever the prompt held, and the in-history and
in-memories recall checks could never come out true. The labels are now
constants, pinned by a test against a prompt the real builder assembled.

The harness also stops at the first sign of failed post-turn work. It checks
/derived and new server.log lines after every turn, keeps its log position
across --resume, and waits for background work to settle before its final
checks. A run with no memories or no summaries now ends "failed", not
"complete".

Both new application tests fail on fec46f6: the lock probe sees
`database is locked`, and memory status stays `idle`. The full backend suite
passes (1392 passed, 17 skipped). A 26-turn re-run against the same host had
0 lock errors, wrote 7 memories and 2 summaries, and used them in the prompt
from turn 8.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0136VBTMUKWYeU6G9HgbDbND

backend/tests/test_context_memory.py

@@ -18,6 +18,7 @@ Two things are asserted throughout rather than assumed:
 """
 
 import asyncio
+import sqlite3
 
 import pytest
 from fastapi import Depends
@@ -26,12 +27,14 @@ from sqlalchemy import select
 
 from app import auth, derived, limits, memorybank, models, summaries
 from app.context import builder, lineage
-from app.database import Base, SessionLocal, engine, get_db
+from app.database import DB_PATH, Base, SessionLocal, engine, get_db
+from app.knowledge import classes
 from app.main import app
 from app.providers import ProviderError
 from app.routers import adventures
 
 from fakes import ScriptedProvider, state_block
+from tools import m11_long_run
 
 
 class StubEmbedder:
@@ -835,3 +838,120 @@ def test_e03_a_summary_generated_after_divergence_carries_no_abandoned_content(c
         assert old[0]["eligible"] is False
     finally:
         mb.summary_provider, mb.embedding_provider = real_summary, real_embed
+
+
+# ------------------------------------------- M11: post-turn work and the write lock
+#
+# Found by the first 26-turn M01 trial on a GPU host. Every turn was accepted,
+# and the run reported "complete" with two memories, no summary and 180
+# `database is locked` errors. A turn that used a memory wrote its use counter
+# before the model call and committed only after the reply. That held SQLite's
+# single write lock for the whole reply. Post-turn memory and summary writes
+# timed out behind it, and the record of each failure timed out the same way.
+
+
+class LockProbe(ScriptedProvider):
+    """A narrator that checks, mid-reply, whether any other writer could get in."""
+
+    seen: list = []
+
+    async def generate(self, parts, *, temperature, max_tokens):
+        # Its own connection, as a post-turn task's session would have. The
+        # short timeout turns "would wait five seconds and fail" into an
+        # immediate answer.
+        probe = sqlite3.connect(DB_PATH, timeout=0.1)
+        try:
+            probe.execute("BEGIN IMMEDIATE")
+            probe.rollback()
+            LockProbe.seen.append("free")
+        except sqlite3.OperationalError as exc:
+            LockProbe.seen.append(str(exc))
+        finally:
+            probe.close()
+        async for item in super().generate(parts, temperature=temperature,
+                                           max_tokens=max_tokens):
+            yield item
+
+
+def test_no_write_lock_is_held_while_the_narrator_is_talking(client, monkeypatch):
+    play(client, "begin", prose="Aldric sets the key down.")
+    memory_id = plant_memory(client, "Aldric hid the ledger beneath the third flagstone.")
+    LockProbe.seen = []
+    monkeypatch.setattr(adventures.turns, "OpenAICompatibleProvider", LockProbe)
+
+    play(client, "I lift the flagstone and look for the ledger.")
+
+    with SessionLocal() as db:
+        # The premise. A turn that retrieved no memory never took the lock, so
+        # the probe below would pass for the wrong reason.
+        assert db.get(models.Memory, memory_id).use_count == 1, (
+            "the turn did not use the planted memory, so this proves nothing")
+    assert LockProbe.seen == ["free"], (
+        "a write transaction was open during the model call, so every "
+        f"post-turn write in that window is locked out: {LockProbe.seen}")
+
+
+def test_a_failed_turn_counts_no_memory_as_used(client, monkeypatch):
+    """The counter is written with the turn now, so a turn that never landed
+    used nothing."""
+    play(client, "begin", prose="Aldric sets the key down.")
+    memory_id = plant_memory(client, "Aldric hid the ledger beneath the third flagstone.")
+    ScriptedProvider.replies = [ProviderError("the narrator is gone")]
+    r = client.post(f"/api/adventures/{client.adv_id}/actions",
+                    json={"type": "do", "text": "I look for the ledger."})
+    assert '"error"' in r.text
+
+    with SessionLocal() as db:
+        assert db.get(models.Memory, memory_id).use_count == 0
+
+
+def test_a_failure_that_breaks_the_session_is_still_recorded(client, monkeypatch):
+    """Recording a failure needs a working session. Without a rollback first,
+    the recorder raised `PendingRollbackError`, the failure went only to the
+    log, and derived status kept reporting a healthy bank."""
+    play(client, "begin", prose="Aldric sets the key down.")
+    existing = plant_memory(client, "Aldric hid the ledger beneath the third flagstone.")
+
+    def collide(adventure, settings, db):
+        # A primary key that already exists: the flush fails and leaves the
+        # session needing a rollback, which is the state a lock timeout on
+        # commit leaves it in.
+        db.add(models.Memory(id=existing, adventure_id=client.adv_id,
+                             text="a second row with the same key"))
+        db.flush()
+
+    monkeypatch.setattr(memorybank, "_evict_over_capacity", collide)
+    asyncio.run(memorybank.run_post_turn(client.adv_id))
+
+    with SessionLocal() as db:
+        rows = {row["kind"]: row for row in derived.report(db, client.adv_id)}
+    assert rows[derived.MEMORY]["status"] == "failed", rows.get(derived.MEMORY)
+    assert "PendingRollbackError" not in rows[derived.MEMORY]["detail"]
+
+
+def test_the_long_run_harness_reads_sections_by_their_real_names(client):
+    """`tools/m11_long_run.py` finds prompt sections by label, and a wrong label
+    is silent: it measured 0 memory tokens and could never find the clue in
+    history or in memories. These are the names the real builder uses."""
+    with SessionLocal() as db:
+        adventure = db.get(models.Adventure, client.adv_id)
+        adventure.authors_note = "Keep the rain in every scene."
+        db.commit()
+    play(client, "begin", prose="Aldric sets the key down.",
+         events=[{"type": "create_entity", "entity": "aldric",
+                  "entity_type": "character", "name": "Aldric"}])
+    for step in range(6):
+        play(client, f"walk on {step}")
+    plant_memory(client, "Aldric hid the ledger beneath the third flagstone.")
+    with SessionLocal() as db:
+        adventure = db.get(models.Adventure, client.adv_id)
+        summaries.record(db, adventure, "The party reached the Crooked Lantern.")
+        db.commit()
+    play(client, "I look for the ledger.")
+
+    labels = {s["label"] for s in context_report(client)["sections"]}
+    for label in (m11_long_run.MEMORIES_LABEL, m11_long_run.SUMMARY_LABEL,
+                  m11_long_run.STATE_LABEL, *m11_long_run.HISTORY_LABELS):
+        assert label in labels, f"the harness reads {label!r}; the prompt has {sorted(labels)}"
+    assert set(m11_long_run.IMPORTED_KNOWLEDGE_LABELS) == {
+        classes.SECTION_ALWAYS_CANON, *classes.CLASS_SECTIONS.values()}

backend/tests/test_embedding_model_switch.py

@@ -138,7 +138,7 @@ def test_retrieval_uses_no_stale_vector_after_the_switch(client, monkeypatch):
         adventure = db.get(models.Adventure, client.adv_id)
         settings = db.query(models.Settings).first()
         result = asyncio.run(
-            memorybank.retrieve_memories(adventure, settings, update_stats=False)
+            memorybank.retrieve_memories(adventure, settings)
         )
         assert result["used"] == []
     finally:

backend/tests/test_m11_long_run_memory.py

@@ -13,6 +13,8 @@ sending it. What they cannot prove is that a hundred turns then fill the bank
 that is what the run itself proves, and `memories_in_bank` in its timeline is
 where it shows.
 """
+import json
+
 import pytest
 from fastapi import Depends
 from fastapi.testclient import TestClient
@@ -182,3 +184,88 @@ def test_a_count_that_cannot_be_read_is_minus_one_not_an_exception(run_for):
     run = run_for(Down())
     run.adv = 7
     assert run.bank_size() == -1
+
+
+# ----------------------------------------------- failed post-turn work stops a run
+
+class Reports:
+    """A server whose derived status, summaries and log say what the test sets."""
+
+    starts = 1
+
+    def __init__(self, log_path, *, status=None, summaries=0, memories=0):
+        self.log_path = log_path
+        self.status = status or []
+        self.summaries = summaries
+        self.memories = memories
+
+    def call(self, method, path, payload=None, timeout=600):
+        if path.endswith("/derived"):
+            return {"status": self.status,
+                    "failing": [r["kind"] for r in self.status if r["status"] == "failed"],
+                    "summaries": [{"id": i} for i in range(self.summaries)]}
+        if path.endswith("/memories"):
+            return [{"id": i} for i in range(self.memories)]
+        return {}
+
+
+def test_a_failed_pass_in_derived_status_stops_the_run(run_for, tmp_path):
+    server = Reports(tmp_path / "server.log", status=[
+        {"kind": "summary", "status": "failed", "detail": "ProviderError: gone"},
+        {"kind": "memory", "status": "idle", "detail": ""},
+    ])
+    run = run_for(server)
+    run.adv = 1
+    found = run.background_failures()
+    assert found == ["summary: ProviderError: gone"]
+
+
+def test_a_failure_the_application_could_not_record_is_found_in_the_log_once(run_for, tmp_path):
+    """The failure that hid the first GPU trial: derived status said `idle` and
+    the only record was in the server log."""
+    log = tmp_path / "server.log"
+    log.write_text("INFO: 200 OK\nERROR:app.memorybank:could not record derived-work failure for 1\n")
+    run = run_for(Reports(log))
+    run.adv = 1
+
+    assert len(run.background_failures()) == 1
+    assert run.background_failures() == [], "the same line was reported twice"
+
+    with log.open("a") as handle:
+        handle.write("ERROR:app.derived:derived summary work failed for adventure 1\n")
+    assert len(run.background_failures()) == 1
+
+
+def test_healthy_status_and_a_quiet_log_find_nothing(run_for, tmp_path):
+    log = tmp_path / "server.log"
+    log.write_text('INFO: "POST /api/adventures/1/actions HTTP/1.1" 200 OK\n')
+    run = run_for(Reports(log, status=[{"kind": "memory", "status": "ok", "detail": ""}]))
+    run.adv = 1
+    assert run.background_failures() == []
+
+
+def test_the_log_position_survives_a_resume(run_for, tmp_path):
+    """Otherwise a resumed run would find the failure that stopped it again, and
+    stop again, however healthy the application now is."""
+    first = run_for(Reports(tmp_path / "server.log"))
+    first.adv, first.log_offset = 1, 4096
+    first.save_resume()
+
+    second = run_for(Reports(tmp_path / "server.log"))
+    second.adopt(json.loads((tmp_path / lr.RESUME_FILE).read_text()))
+    assert second.log_offset == 4096
+
+
+def test_a_run_with_no_summary_or_no_memory_is_not_complete(run_for, tmp_path):
+    log = tmp_path / "server.log"
+    assert "summaries=0" in lr._activation_shortfall(
+        _with_adv(run_for(Reports(log, memories=3, summaries=0))))
+    assert "memories_in_bank=0" in lr._activation_shortfall(
+        _with_adv(run_for(Reports(log, memories=0, summaries=2))))
+    assert lr._activation_shortfall(
+        _with_adv(run_for(Reports(log, memories=3, summaries=1)))) is None
+
+
+def _with_adv(run):
+    run.adv = 1
+    return run

backend/tests/test_memory_nodes.py

@@ -196,7 +196,7 @@ def restore_embedding_provider():
 def retrieved(adventure, settings) -> set[str]:
     memorybank.embedding_provider = lambda s: StubEmbedder()
     result = asyncio.run(
-        memorybank.retrieve_memories(adventure, settings, update_stats=False)
+        memorybank.retrieve_memories(adventure, settings)
     )
     assert result["error"] is None, result["error"]
     return {m["text"] for m in result["used"]}

backend/tests/test_memory_retrieval.py

@@ -121,7 +121,6 @@ def bank(db, adventure):
 
 def retrieve(adventure, settings, embedder, **kwargs):
     memorybank.embedding_provider = lambda s: embedder
-    kwargs.setdefault("update_stats", False)
     return asyncio.run(memorybank.retrieve_memories(adventure, settings, **kwargs))
 
 
@@ -185,10 +184,11 @@ def test_missing_embedding_model_is_reported(db, adventure, settings, bank):
     assert result["used"] == [] and "embedding model" in result["error"]
 
 
-def test_update_stats_bumps_only_the_used(db, adventure, settings, bank):
+def test_record_use_bumps_only_the_used(db, adventure, settings, bank):
     settings.memory_top_k = 1
     db.commit()
-    retrieve(adventure, settings, StubEmbedder((1.0, 0.0, 0.0)), update_stats=True)
+    used = retrieve(adventure, settings, StubEmbedder((1.0, 0.0, 0.0)))
+    memorybank.record_use(db, used)
     db.commit()
     db.expire_all()
 
@@ -200,7 +200,7 @@ def test_update_stats_bumps_only_the_used(db, adventure, settings, bank):
 def test_dry_runs_do_not_bump_the_counters(db, adventure, settings, bank):
     """Insights assembles a context without spending a turn. It must not
     look like the memories were used."""
-    retrieve(adventure, settings, StubEmbedder(), update_stats=False)
+    retrieve(adventure, settings, StubEmbedder())
     db.commit()
     db.expire_all()
     assert all(db.get(models.Memory, m.id).use_count == 0 for m in bank.values())