Replace clustering with a catalogue, and hand-write the references to judge it against

The pipeline's embed-and-cluster step is dead, and this commit holds both the
evidence for that and the step proposed to replace it.

Predicaments. Scenes are re-described as "what the person is up against", with
no names, jobs or places, then embedded and clustered (redescribe.py,
topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading
both side by side. Two defects the pilot exposed are fixed: split.py missed
titles in quotes and a contents subtitle after a dash, so three stories had been
merged into their neighbours, and strip_names.py read New York place names as
people. The corrected corpus is probe/v2 (97 stories, 839 scenes);
carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from
20% to 13% at k=60, short of the pre-registered 10%.

Hand references. Three corpora were read scene by scene and written up by hand,
under the same prompt rules the local models get, as a baseline to judge them
against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's
Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of
the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a
readable page. No inference was used for any of them.

Catalogue. probe/catalogue maps every hand group in the three references onto 36
situation entries, with an answer key per corpus and one recurrence rule applied
to all three. classify.py assigns a scene one entry or none, leave-one-corpus-
out; score.py checks it against the key, with a self-test on random labels.

Why clustering is out: hand-written predicaments, embedded and clustered exactly
as the model's were, agree with the hand grouping at ARI 0.05 — no better than
the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even
shortlist: the hand label is the nearest entry 13% of the time and in the top 8
half the time.

The classification runs are not here. The dev and test runs are pre-registered
in probe/catalogue/README.md with the bar set beforehand, and are blocked on the
inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene
partial output in out/ is not a result.

Review page. The situation review is now a browser page rather than JSON edited
by hand (review_page.py, review_page_logic.cjs with Node tests, format schema
v2). It has never been rendered in a real browser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1

tools/story-to-pack/probe/wharton/claude_export.py

@@ -0,0 +1,41 @@
+"""Assemble the hand-written reference (claude/batches/*.json) into files the probe's scripts read,
+and measure the hand groups with the probe's own topic-share rule.
+
+  cd probe/wharton && python3 claude_export.py
+
+Writes claude/summaries.json and claude/predicaments.json in the {"summaries": [{chunk, summary}]}
+shape that embed.py, topic_share.py and groups_page.py take, so a local-inference run on this corpus
+can be set beside it scene for scene. Then reports, for each group in claude/groups.json, the share
+of its scenes whose summary contains the group's most common job, relationship or place word
+(topic_share.py's measure, applied to hand groups instead of k-means groups), and how many
+predicament sentences leak such a word. No inference.
+"""
+import collections, glob, json, pathlib, re, statistics, sys
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent))
+from topic_words import TOPIC_WORDS
+
+rows = []
+for f in sorted(glob.glob('claude/batches/s*.json')):
+    rows += json.loads(pathlib.Path(f).read_text(encoding='utf-8'))
+chunks = json.loads(pathlib.Path('chunks.json').read_text(encoding='utf-8'))
+assert [r['chunk'] for r in rows] == list(range(len(chunks)))
+for key, out in (('summary', 'summaries.json'), ('predicament', 'predicaments.json')):
+    pathlib.Path('claude', out).write_text(json.dumps(
+        {'source': 'hand-written by Claude, reading each scene', 'field': key,
+         'summaries': [{'chunk': r['chunk'], 'summary': r[key]} for r in rows]},
+        indent=1, ensure_ascii=False), encoding='utf-8')
+
+words = lambda t: set(re.findall(r"[a-zé]+", t.lower())) & TOPIC_WORDS
+summ = {r['chunk']: words(r['summary']) for r in rows}
+groups = json.loads(pathlib.Path('claude/groups.json').read_text(encoding='utf-8'))['groups']
+shares = []
+for g in groups:
+    m = g['members']
+    top = collections.Counter(w for i in m for w in summ[i]).most_common(1)
+    word, hits = top[0] if top else ('-', 0)
+    shares.append(hits / len(m))
+    print(f"{g['id']:18} {len(m):2} scenes  top topic word {word!r:14} in {hits / len(m):4.0%}")
+print(f'\nmedian top topic-word share over {len(groups)} hand groups: {statistics.median(shares):.0%}')
+leak = [r for r in rows if words(r['predicament'])]
+print(f'predicaments containing a topic word: {len(leak)} of {len(rows)} '
+      f'({collections.Counter(w for r in leak for w in words(r["predicament"])).most_common(8)})')