Replace clustering with a catalogue, and hand-write the references to judge it against

The pipeline's embed-and-cluster step is dead, and this commit holds both the
evidence for that and the step proposed to replace it.

Predicaments. Scenes are re-described as "what the person is up against", with
no names, jobs or places, then embedded and clustered (redescribe.py,
topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading
both side by side. Two defects the pilot exposed are fixed: split.py missed
titles in quotes and a contents subtitle after a dash, so three stories had been
merged into their neighbours, and strip_names.py read New York place names as
people. The corrected corpus is probe/v2 (97 stories, 839 scenes);
carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from
20% to 13% at k=60, short of the pre-registered 10%.

Hand references. Three corpora were read scene by scene and written up by hand,
under the same prompt rules the local models get, as a baseline to judge them
against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's
Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of
the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a
readable page. No inference was used for any of them.

Catalogue. probe/catalogue maps every hand group in the three references onto 36
situation entries, with an answer key per corpus and one recurrence rule applied
to all three. classify.py assigns a scene one entry or none, leave-one-corpus-
out; score.py checks it against the key, with a self-test on random labels.

Why clustering is out: hand-written predicaments, embedded and clustered exactly
as the model's were, agree with the hand grouping at ARI 0.05 — no better than
the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even
shortlist: the hand label is the nearest entry 13% of the time and in the top 8
half the time.

The classification runs are not here. The dev and test runs are pre-registered
in probe/catalogue/README.md with the bar set beforehand, and are blocked on the
inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene
partial output in out/ is not a result.

Review page. The situation review is now a browser page rather than JSON edited
by hand (review_page.py, review_page_logic.cjs with Node tests, format schema
v2). It has never been rendered in a real browser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
This commit is contained in:
JesseMarkowitz
2026-09-20 17:19:42 -04:00
co-authored by Claude Opus 5
parent fa3769d0fe
commit e5617b86ba
135 changed files with 82027 additions and 1778 deletions
+442
View File
@@ -0,0 +1,442 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Situation review · __RUN_ID__</title>
<style>
:root {
--bg: #f7f6f2; --card: #ffffff; --ink: #1d1d1b; --muted: #66655f; --line: #dedcd4;
--accent: #2f5d8a; --accent-ink: #ffffff; --good: #2e7d4f; --warn: #9a5a00; --bad: #b3261e;
--chip: #e9eef5; --struck: #9a9890;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #161614; --card: #211f1c; --ink: #ecebe6; --muted: #a6a49c; --line: #3a3934;
--accent: #86b4e2; --accent-ink: #0d1b28; --good: #74cf99; --warn: #f0b64d; --bad: #f28b82;
--chip: #2a3440; --struck: #77756e;
}
}
* { box-sizing: border-box; }
body { margin: 0; background: var(--bg); color: var(--ink);
font: 16px/1.55 system-ui, -apple-system, "Segoe UI", Roboto, sans-serif; }
header.top { position: sticky; top: 0; z-index: 2; background: var(--card);
border-bottom: 1px solid var(--line); padding: 10px 16px; }
.wrap { max-width: 900px; margin: 0 auto; padding: 16px; }
.row { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; }
.spacer { flex: 1; }
h1 { font-size: 1.05rem; margin: 0; }
h2 { font-size: 1.35rem; margin: 0; }
.muted { color: var(--muted); }
.pills { display: flex; flex-wrap: wrap; gap: 4px; margin-top: 8px; align-items: center; }
.pill { border: 1px solid var(--line); background: var(--card); color: var(--ink); border-radius: 999px;
padding: 1px 10px; font: inherit; font-size: .85rem; cursor: pointer; min-width: 2.2em; }
.pill.current { outline: 2px solid var(--accent); outline-offset: 1px; }
.pill.done { background: var(--good); border-color: var(--good); color: var(--card); }
.pill.problem { background: var(--warn); border-color: var(--warn); color: var(--card); }
.pill.skipped { color: var(--muted); }
.card { background: var(--card); border: 1px solid var(--line); border-radius: 10px; padding: 16px 18px; margin: 14px 0; }
.btn { font: inherit; border: 1px solid var(--line); background: var(--card); color: var(--ink);
border-radius: 8px; padding: 7px 14px; cursor: pointer; }
.btn.primary { background: var(--accent); border-color: var(--accent); color: var(--accent-ink); }
.btn.on { background: var(--accent); border-color: var(--accent); color: var(--accent-ink); }
.btn:disabled { opacity: .45; cursor: not-allowed; }
.q { font-weight: 650; margin: 0 0 10px; }
.tip { font-size: .92rem; color: var(--muted); margin: 8px 0 0; }
.status { font-weight: 600; }
.status.problem { color: var(--bad); }
.status.done { color: var(--good); }
ul.scenes { list-style: none; margin: 0; padding: 0; border-top: 1px solid var(--line); }
ul.scenes li { display: flex; gap: 12px; padding: 8px 2px; border-bottom: 1px solid var(--line); }
ul.scenes li.out .text { text-decoration: line-through; color: var(--struck); }
ul.scenes input { width: 20px; height: 20px; margin-top: 3px; flex: none; cursor: pointer; }
.story { display: block; font-size: .8rem; color: var(--muted); }
.shape { margin: 10px 0 0; font-weight: 600; }
.shape.ok { color: var(--good); }
.shape.bad { color: var(--bad); }
.name { display: flex; flex-wrap: wrap; gap: 8px; align-items: center; font-size: 1.05rem; }
.name input { font: inherit; padding: 6px 8px; border: 1px solid var(--line); border-radius: 6px;
background: var(--bg); color: var(--ink); width: 15em; max-width: 100%; }
.chips { display: flex; flex-wrap: wrap; gap: 6px; margin-top: 10px; align-items: center; }
.chip { font: inherit; border: none; background: var(--chip); color: var(--ink); border-radius: 6px;
padding: 2px 9px; cursor: pointer; font-size: .9rem; }
.preview { margin: 12px 0 0; font-size: 1.05rem; }
select { font: inherit; padding: 6px; border-radius: 6px; border: 1px solid var(--line);
background: var(--bg); color: var(--ink); max-width: 100%; }
.example { border-left: 3px solid var(--line); padding: 2px 14px; margin: 12px 0; }
.example ul { margin: 4px 0; }
table.summary { width: 100%; border-collapse: collapse; }
table.summary td { border-bottom: 1px solid var(--line); padding: 6px 4px; vertical-align: top; }
pre { background: var(--bg); border: 1px solid var(--line); border-radius: 6px; padding: 8px 10px; overflow-x: auto; }
.saved { font-size: .85rem; color: var(--muted); }
kbd { border: 1px solid var(--line); border-radius: 4px; padding: 0 5px; font-size: .85rem; }
</style>
</head>
<body>
<header class="top">
<div class="row" id="topbar"></div>
<div id="pills"></div>
</header>
<main class="wrap" id="app"></main>
<noscript><p class="wrap">This review page needs JavaScript turned on.</p></noscript>
<script>
/*__LOGIC__*/
</script>
<script>
const DOC = /*__DATA__*/null;
(() => {
const L = ReviewLogic;
const run = DOC.run;
const order = L.orderedIds(DOC);
const KEY = `stp-review:${run.id}`;
let saveNote = '';
let lastRoleField = null;
let state = load() || { entries: L.initialEntries(DOC), view: 'intro', current: 0 };
// ---- saving -------------------------------------------------------------------------
function load() {
try {
const raw = localStorage.getItem(KEY);
if (!raw) return null;
const saved = JSON.parse(raw);
const restored = L.fromReviewFile(DOC, saved.file);
if (restored.error) return null;
saveNote = 'Picked up where you left off.';
const view = ['intro', 'candidate', 'finish'].includes(saved.view) ? saved.view : 'candidate';
const current = Math.min(Math.max(0, Number(saved.current) || 0), order.length - 1);
return { entries: restored.entries, view, current };
} catch (_) {
return null;
}
}
function persist() {
try {
const file = L.toReviewFile(DOC, state.entries, new Date().toISOString());
localStorage.setItem(KEY, JSON.stringify({ file, view: state.view, current: state.current }));
saveNote = 'Progress saved in this browser.';
} catch (_) {
saveNote = 'This browser is not keeping progress: use “Save review file” before closing.';
}
}
function update(change, { top = false } = {}) {
change();
persist();
render();
if (top) window.scrollTo(0, 0);
}
function download() {
const file = L.toReviewFile(DOC, state.entries, new Date().toISOString());
const blob = new Blob([JSON.stringify(file, null, 1)], { type: 'application/json' });
const link = document.createElement('a');
link.href = URL.createObjectURL(blob);
link.download = `review-${run.id}.json`;
document.body.appendChild(link);
link.click();
link.remove();
setTimeout(() => URL.revokeObjectURL(link.href), 2000);
saveNote = `Saved review-${run.id}.json.`;
renderTop();
}
function upload(event) {
const chosen = event.target.files && event.target.files[0];
if (!chosen) return;
const reader = new FileReader();
reader.onload = () => {
let data;
try { data = JSON.parse(reader.result); } catch (_) { alert('That file is not a review file.'); return; }
const restored = L.fromReviewFile(DOC, data);
if (restored.error) { alert(restored.error); return; }
if (!confirm('Replace what is on this page with the review in that file?')) return;
update(() => { state.entries = restored.entries; state.view = 'finish'; });
};
reader.readAsText(chosen);
}
// ---- building the page ----------------------------------------------------------------
function el(tag, attrs, ...children) {
const node = document.createElement(tag);
for (const [key, value] of Object.entries(attrs || {})) {
if (value === null || value === undefined || value === false) continue;
if (key === 'class') node.className = value;
else if (key.startsWith('on')) node.addEventListener(key.slice(2), value);
else if (key === 'value') node.value = value;
else if (key === 'checked') node.checked = !!value;
else node.setAttribute(key, value === true ? '' : value);
}
for (const child of children.flat()) {
if (child === null || child === undefined || child === false) continue;
node.append(child instanceof Node ? child : document.createTextNode(String(child)));
}
return node;
}
const entryOf = (id) => state.entries.find((e) => e.id === id);
const go = (index) => update(() => { state.view = 'candidate'; state.current = index; }, { top: true });
function render() {
renderTop();
const app = document.getElementById('app');
if (state.view === 'intro') app.replaceChildren(intro());
else if (state.view === 'finish') app.replaceChildren(finish());
else app.replaceChildren(candidateView(order[state.current]));
}
function renderTop() {
const p = L.progress(DOC, state.entries);
document.getElementById('topbar').replaceChildren(
el('h1', {}, `Situation review · ${run.id}`),
el('span', { class: 'muted' },
`${p.decided} of ${p.total} groups decided${p.problems ? ` · ${p.problems} need attention` : ''}`),
el('span', { class: 'spacer' }),
el('span', { class: 'saved' }, saveNote),
el('button', { class: 'btn', onclick: () => update(() => { state.view = 'intro'; }, { top: true }) }, 'How this works'),
el('button', { class: 'btn primary', onclick: () => update(() => { state.view = 'finish'; }, { top: true }) }, 'Finish & save'),
);
const pill = (id) => {
const index = order.indexOf(id);
const s = L.status(DOC, state.entries, id);
const current = state.view === 'candidate' && state.current === index;
return el('button', {
class: `pill ${s.state}${current ? ' current' : ''}`,
title: `${L.displayName(DOC, id)}: ${s.message}`,
onclick: () => go(index),
}, L.shortName(DOC, id));
};
const accepted = order.filter((id) => DOC.candidates[id].kind === 'accepted');
const extras = order.filter((id) => DOC.candidates[id].kind === 'borderline');
document.getElementById('pills').replaceChildren(el('div', { class: 'pills' },
el('span', { class: 'muted' }, 'Groups:'),
accepted.map(pill),
extras.length ? el('span', { class: 'muted' }, ' Optional extras:') : null,
extras.map(pill),
));
}
function example(verdict, lines, note) {
return el('div', { class: 'example' },
el('strong', {}, verdict),
el('ul', {}, lines.map((line) => el('li', {}, line))),
el('div', { class: 'muted' }, note));
}
function intro() {
const p = L.progress(DOC, state.entries);
return el('div', {},
el('div', { class: 'card' },
el('h2', {}, 'What you are doing'),
el('p', {},
`The program read ${run.scenes} scenes from ${run.stories} stories, summarised each in a sentence, ` +
`and grouped scenes that seem alike. ${p.total} groups might each be one recurring situation: ` +
'something that could come up again and again in the game, with different people. ' +
'Your job is to check each group.'),
el('p', {}, 'Each group asks you three things, one at a time:'),
el('ol', {},
el('li', {}, el('strong', {}, 'Is it one situation? '), 'Keep it, drop it, or mark it the same as a group you already kept.'),
el('li', {}, el('strong', {}, 'Which scenes don’t fit? '), 'Untick them. Every scene stays in unless you untick it.'),
el('li', {}, el('strong', {}, 'What is it? '), 'Fill in one short sentence: “A ___ wants ___ from ___.”')),
el('p', {},
'Expect to drop roughly a quarter of the groups. Allow about a minute each. ' +
'Everything saves in this browser as you go, so you can stop and come back. ' +
'At the end, “Finish & save” saves a review file for the next step.'),
),
el('div', { class: 'card' },
el('h2', {}, 'What counts as a situation'),
el('p', {}, 'One person wanting something specific from another person — the kind of moment that could come round again with different people in it.'),
example('Keep', [
'A lodger begs the landlady for one more week.',
'A tenant hides on the stairs on rent day.',
'A landlady tells a family they are out on Monday.',
], 'Name it: “A lodger wants more time to pay from a landlady.”'),
example('Drop: a shared place, not a situation', [
'Two diners argue about politics.',
'A waiter carries six plates at once.',
'A cook burns the soup.',
], 'They all happen in a restaurant, but nobody wants the same kind of thing from anybody.'),
example('Drop: too vague to name', [
'Someone is tense with someone.',
'A conflict over a personal matter.',
'Power shifts between two people.',
], 'If the best you can say is “conflict” or “people want things”, drop it.'),
example('Same as another group', [
'Group 2 is suitors pressing women to marry them.',
'Group 7 is a wealthy man proposing and being turned down.',
], 'Keep Group 2. On Group 7, choose “Same situation as… Group 2”. Its scenes join Group 2, and it needs no name of its own.'),
el('p', { class: 'tip' }, 'These examples are made up, to show the idea.'),
),
el('div', { class: 'row' },
el('button', { class: 'btn primary', onclick: () => update(() => { state.view = 'candidate'; }, { top: true }) },
p.decided ? 'Carry on reviewing' : 'Start with Group 1'),
el('span', { class: 'muted' }, 'Shortcuts while reviewing: ', el('kbd', {}, 'K'), ' keep · ', el('kbd', {}, 'D'), ' drop · ',
el('kbd', {}, '→'), ' next · ', el('kbd', {}, '←'), ' back')),
);
}
function roleField(entry, key, placeholder) {
return el('input', {
value: entry[key], placeholder, maxlength: DOC.rules.maxFieldChars, 'aria-label': placeholder,
onfocus: () => { if (key !== 'wants') lastRoleField = key; },
oninput: (event) => {
entry[key] = event.target.value;
persist();
renderTop();
const preview = document.getElementById('preview');
if (preview) preview.textContent = L.sentence(entry);
},
});
}
function fillRole(entry, role) {
const target = lastRoleField && !entry[lastRoleField].trim()
? lastRoleField
: (!entry.actor.trim() ? 'actor' : (!entry.counterpart.trim() ? 'counterpart' : (lastRoleField || 'actor')));
update(() => {
entry[target] = target === 'counterpart' ? `a ${role}` : role;
lastRoleField = target === 'actor' ? 'counterpart' : 'actor';
});
}
function candidateView(id) {
const c = DOC.candidates[id];
const e = entryOf(id);
const index = order.indexOf(id);
const s = L.status(DOC, state.entries, id);
const set = (patch) => update(() => Object.assign(e, patch));
const inGroup = e.decision === 'keep' || e.decision === 'same';
const targets = L.keptTargets(DOC, state.entries, id);
const excluded = new Set(e.excluded);
const shape = L.shapeCheck(L.groupMembers(DOC, state.entries, e.decision === 'same' && e.same_as ? e.same_as : id), DOC.rules);
const decisions = el('div', { class: 'row' },
el('button', { class: `btn${e.decision === 'keep' ? ' on' : ''}`, onclick: () => set({ decision: 'keep', same_as: null }) },
'Yes, keep it'),
c.kind === 'accepted'
? el('button', { class: `btn${e.decision === 'drop' ? ' on' : ''}`, onclick: () => set({ decision: 'drop', same_as: null }) }, 'No, drop it')
: el('button', { class: `btn${e.decision === 'skip' ? ' on' : ''}`, onclick: () => set({ decision: 'skip', same_as: null }) }, 'Leave it out'),
el('span', { class: 'muted' }, 'or'),
el('select', {
disabled: targets.length === 0 && e.decision !== 'same',
onchange: (event) => set(event.target.value
? { decision: 'same', same_as: event.target.value }
: { decision: c.kind === 'accepted' ? null : 'skip', same_as: null }),
},
el('option', { value: '' }, targets.length ? 'Same situation as…' : 'Same situation as… (keep a group first)'),
targets.map((t) => el('option', { value: t, selected: e.decision === 'same' && e.same_as === t },
`${L.displayName(DOC, t)}: ${L.sentence(entryOf(t))}`))),
);
const scenes = el('ul', { class: 'scenes' }, c.members.map((m) => {
const out = excluded.has(m.member);
return el('li', { class: out ? 'out' : '' },
el('input', {
type: 'checkbox', checked: !out, disabled: !inGroup, 'aria-label': 'This scene fits',
onchange: (event) => set({
excluded: event.target.checked ? e.excluded.filter((x) => x !== m.member) : [...e.excluded, m.member],
}),
}),
el('div', { class: 'text' }, m.summary, el('span', { class: 'story' }, m.title)));
}));
let third = null;
if (e.decision === 'keep') {
third = el('div', { class: 'card' },
el('p', { class: 'q' }, '3. What is it?'),
el('div', { class: 'name' },
'A', roleField(e, 'actor', 'who wants something'),
'wants', roleField(e, 'wants', 'what they want'),
'from', roleField(e, 'counterpart', 'whom'), '.'),
c.role_hints && c.role_hints.length
? el('div', { class: 'chips' }, el('span', { class: 'muted' }, 'People in these scenes (click to use):'),
c.role_hints.map((role) => el('button', { class: 'chip', onclick: () => fillRole(e, role) }, role)))
: null,
el('p', { class: 'preview', id: 'preview' }, L.sentence(e)),
el('p', { class: 'tip' },
'Use roles, not names: “a lodger”, “her husband”, “a wealthy suitor”. ' +
'If you can’t say what they want, that usually means the group should be dropped.'));
} else if (e.decision === 'same' && e.same_as) {
third = el('div', { class: 'card' },
el('p', {}, `Merged into ${L.displayName(DOC, e.same_as)}: “${L.sentence(entryOf(e.same_as))}”`),
el('p', { class: 'tip' }, 'Its ticked scenes join that group, so it needs no name of its own.'));
}
return el('div', {},
el('div', { class: 'row' },
el('h2', {}, L.displayName(DOC, id)),
el('span', { class: 'muted' }, `${c.members.length} scenes from ${c.stats.stories} stories`),
el('span', { class: 'spacer' }),
el('span', { class: `status ${s.state}` }, s.message)),
c.kind === 'borderline'
? el('p', { class: 'tip' }, 'Optional extra. The program nearly left this group out, so leave it out unless it clearly shows one situation.')
: null,
el('div', { class: 'card' },
el('p', { class: 'q' }, '1. Do most of these scenes show one situation: one person wanting something specific from another?'),
decisions,
el('p', { class: 'tip' }, 'Read the scenes below first. A shared place, job, mood or word is not enough.')),
el('div', { class: 'card' },
el('p', { class: 'q' }, inGroup ? '2. Untick any scene that doesn’t show it.' : 'The scenes'),
scenes,
inGroup ? el('p', { class: `shape ${shape.ok ? 'ok' : 'bad'}` }, shape.text) : null),
third,
el('div', { class: 'row' },
el('button', { class: 'btn', disabled: index === 0, onclick: () => go(index - 1) }, '← Back'),
el('span', { class: 'spacer' }),
index < order.length - 1
? el('button', { class: 'btn primary', onclick: () => go(index + 1) }, 'Next →')
: el('button', { class: 'btn primary', onclick: () => update(() => { state.view = 'finish'; }, { top: true }) }, 'Finish & save')),
);
}
function finish() {
const p = L.progress(DOC, state.entries);
const ready = p.decided === p.total && p.problems === 0;
const rows = order.map((id) => {
const s = L.status(DOC, state.entries, id);
return el('tr', {},
el('td', {}, el('button', { class: `pill ${s.state}`, onclick: () => go(order.indexOf(id)) }, L.displayName(DOC, id))),
el('td', {}, s.message));
});
return el('div', {},
el('div', { class: 'card' },
el('h2', {}, ready ? 'Ready to save' : 'Not finished yet'),
el('p', {}, ready
? 'Every group is decided and nothing needs attention.'
: `${p.total - p.decided} groups not decided${p.problems ? `, ${p.problems} need attention` : ''}. ` +
'You can still save now: the file keeps your progress, and the next step lists what is missing.'),
el('div', { class: 'row' },
el('button', { class: 'btn primary', onclick: download }, 'Save review file'),
el('label', { class: 'btn' }, 'Load a review file',
el('input', { type: 'file', accept: '.json,application/json', style: 'display:none', onchange: upload }))),
el('p', { class: 'tip' },
`The browser saves review-${run.id}.json, usually into Downloads. Then, in the story-to-pack probe folder, run:`),
el('pre', {}, `python3 apply_review.py review/${run.id}`),
el('p', { class: 'tip' }, 'It finds the file in Downloads by itself, checks it, and writes situations.json.')),
el('div', { class: 'card' }, el('table', { class: 'summary' }, rows)),
);
}
document.addEventListener('keydown', (event) => {
if (state.view !== 'candidate' || event.ctrlKey || event.metaKey || event.altKey) return;
// Never while typing a name or choosing from the list.
const tag = (event.target.tagName || '').toLowerCase();
if (['input', 'select', 'textarea'].includes(tag)) return;
const id = order[state.current];
const e = entryOf(id);
const kind = DOC.candidates[id].kind;
if (event.key === 'k' || event.key === 'K') update(() => { e.decision = 'keep'; e.same_as = null; });
else if (event.key === 'd' || event.key === 'D') update(() => { e.decision = kind === 'accepted' ? 'drop' : 'skip'; e.same_as = null; });
else if (event.key === 'ArrowRight' && state.current < order.length - 1) go(state.current + 1);
else if (event.key === 'ArrowLeft' && state.current > 0) go(state.current - 1);
});
render();
})();
</script>
</body>
</html>