Files
TheLadder/tools/story-to-pack/probe/v2/claude/report_template.html
T
JesseMarkowitzandClaude Opus 5 e5617b86ba Replace clustering with a catalogue, and hand-write the references to judge it against
The pipeline's embed-and-cluster step is dead, and this commit holds both the
evidence for that and the step proposed to replace it.

Predicaments. Scenes are re-described as "what the person is up against", with
no names, jobs or places, then embedded and clustered (redescribe.py,
topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading
both side by side. Two defects the pilot exposed are fixed: split.py missed
titles in quotes and a contents subtitle after a dash, so three stories had been
merged into their neighbours, and strip_names.py read New York place names as
people. The corrected corpus is probe/v2 (97 stories, 839 scenes);
carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from
20% to 13% at k=60, short of the pre-registered 10%.

Hand references. Three corpora were read scene by scene and written up by hand,
under the same prompt rules the local models get, as a baseline to judge them
against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's
Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of
the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a
readable page. No inference was used for any of them.

Catalogue. probe/catalogue maps every hand group in the three references onto 36
situation entries, with an answer key per corpus and one recurrence rule applied
to all three. classify.py assigns a scene one entry or none, leave-one-corpus-
out; score.py checks it against the key, with a self-test on random labels.

Why clustering is out: hand-written predicaments, embedded and clustered exactly
as the model's were, agree with the hand grouping at ARI 0.05 — no better than
the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even
shortlist: the hand label is the nearest entry 13% of the time and in the top 8
half the time.

The classification runs are not here. The dev and test runs are pre-registered
in probe/catalogue/README.md with the bar set beforehand, and are blocked on the
inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene
partial output in out/ is not a result.

Review page. The situation review is now a browser page rather than JSON edited
by hand (review_page.py, review_page_logic.cjs with Node tests, format schema
v2). It has never been rendered in a real browser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
2026-09-20 17:19:42 -04:00

258 lines
18 KiB
HTML

<title>O. Henry Reference Pass</title>
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Old+Standard+TT:ital,wght@0,400;0,700;1,400&family=IBM+Plex+Sans:ital,wght@0,400;0,500;0,600;1,400&family=IBM+Plex+Mono:wght@400;500&display=swap">
<style>
:root{
--ground:#F3F5F2; --surface:#FFFFFF; --sunk:#E9EDE9; --ink:#1B2320; --muted:#56615C; --faint:#8A948F;
--rule:#D5DCD7; --accent:#1F6F5C; --accent-soft:#DCEDE6; --brass:#8C6C22; --brass-soft:#F2E8CF;
--up:#2E7A48; --down:#A5432D; --chip:#EEF2EF;
--display:"Old Standard TT", "Iowan Old Style", Georgia, serif;
--body:"IBM Plex Sans", system-ui, -apple-system, "Segoe UI", sans-serif;
--mono:"IBM Plex Mono", ui-monospace, "SFMono-Regular", Menlo, monospace;
}
@media (prefers-color-scheme: dark){
:root:not([data-theme="light"]){
--ground:#111614; --surface:#18201C; --sunk:#1F2924; --ink:#E3E9E5; --muted:#A5B1AB; --faint:#78847E;
--rule:#2C3833; --accent:#62C2A6; --accent-soft:#1D3A31; --brass:#D5B561; --brass-soft:#3A3120;
--up:#6CC48C; --down:#E08A74; --chip:#223029;
}
}
:root[data-theme="dark"]{
--ground:#111614; --surface:#18201C; --sunk:#1F2924; --ink:#E3E9E5; --muted:#A5B1AB; --faint:#78847E;
--rule:#2C3833; --accent:#62C2A6; --accent-soft:#1D3A31; --brass:#D5B561; --brass-soft:#3A3120;
--up:#6CC48C; --down:#E08A74; --chip:#223029;
}
*{box-sizing:border-box}
body{background:var(--ground); color:var(--ink); font:15px/1.55 var(--body); padding-inline:16px; padding-block:0 64px}
.wrap{max-width:1040px; margin:0 auto}
header.top{padding-block:40px 20px; border-bottom:1px solid var(--rule)}
.eyebrow{font:500 11px/1 var(--mono); letter-spacing:.12em; text-transform:uppercase; color:var(--accent)}
h1{font:400 clamp(30px,5vw,44px)/1.1 var(--display); margin:.35em 0 .3em; text-wrap:balance}
h2{font:400 28px/1.2 var(--display); margin:0 0 .4em; text-wrap:balance}
h3{font:600 17px/1.3 var(--body); margin:0; text-wrap:balance}
p{max-width:68ch; margin:.5em 0}
.lede{font-size:16.5px; color:var(--muted); max-width:70ch}
nav.jump{position:sticky; top:env(safe-area-inset-top,0px); z-index:5; background:var(--ground); border-bottom:1px solid var(--rule);
display:flex; gap:4px; flex-wrap:wrap; padding-block:8px}
nav.jump a{font:500 13px/1 var(--body); color:var(--muted); text-decoration:none; padding:8px 12px; border-radius:6px}
nav.jump a:hover,nav.jump a:focus-visible{color:var(--ink); background:var(--sunk); outline:none}
section{padding-block:36px 8px}
.facts{display:grid; grid-template-columns:repeat(auto-fit,minmax(190px,1fr)); gap:12px; margin-top:18px}
.fact{background:var(--surface); border:1px solid var(--rule); border-radius:8px; padding:14px 16px}
.fact b{display:block; font:400 28px/1.1 var(--display); font-variant-numeric:tabular-nums}
.fact span{color:var(--muted); font-size:13px}
.note{background:var(--brass-soft); border-left:3px solid var(--brass); padding:12px 16px; border-radius:0 6px 6px 0; margin:16px 0; max-width:none}
.note p{margin:.3em 0}
code,.mono{font-family:var(--mono); font-size:12.5px}
.tablewrap{overflow-x:auto; border:1px solid var(--rule); border-radius:8px; background:var(--surface); margin:12px 0}
table{border-collapse:collapse; width:100%; font-size:13.5px}
th,td{text-align:left; padding:8px 12px; border-bottom:1px solid var(--rule); vertical-align:top}
th{font:600 11px/1.3 var(--mono); letter-spacing:.06em; text-transform:uppercase; color:var(--muted); background:var(--sunk)}
td.num,th.num{text-align:right; font-variant-numeric:tabular-nums}
tr:last-child td{border-bottom:none}
/* situations */
.sitlist{display:flex; flex-direction:column; gap:14px; margin-top:16px}
details.sit{background:var(--surface); border:1px solid var(--rule); border-radius:10px}
details.sit>summary{list-style:none; cursor:pointer; padding:16px 18px; display:grid; grid-template-columns:1fr auto; gap:6px 16px; align-items:start}
details.sit>summary::-webkit-details-marker{display:none}
details.sit>summary:focus-visible{outline:2px solid var(--accent); outline-offset:2px; border-radius:10px}
.sit .sentence{grid-column:1/-1; color:var(--muted); font-size:14.5px; margin:0}
.sit .meta{display:flex; gap:6px; flex-wrap:wrap; justify-content:flex-end}
.pill{font:500 11px/1 var(--mono); padding:5px 8px; border-radius:999px; background:var(--chip); color:var(--muted); white-space:nowrap}
.pill.ok{background:var(--accent-soft); color:var(--accent)}
.pill.below{background:var(--brass-soft); color:var(--brass)}
.sitbody{padding:0 18px 18px; border-top:1px solid var(--rule)}
.stories{font-size:13px; color:var(--muted); margin:12px 0 4px}
.stories i{font-style:normal; font-variant:small-caps; letter-spacing:.02em; color:var(--ink)}
.event{margin-top:18px}
.event h4{font:400 20px/1.25 var(--display); margin:0 0 4px}
.event .ev-meta{font-size:12.5px; color:var(--muted)}
.descs{display:flex; flex-direction:column; gap:8px; margin:10px 0 14px; padding:0; list-style:none}
.descs li{background:var(--sunk); border-radius:6px; padding:10px 14px; font-style:italic}
.descs li .src{font-style:normal; display:block; margin-top:4px}
.src{font:400 11.5px/1.3 var(--mono); color:var(--faint)}
.opts{display:grid; grid-template-columns:1fr; gap:10px}
.opt{border:1px solid var(--rule); border-radius:8px; padding:10px 14px; display:grid; gap:6px}
.opt .label{font-weight:600}
.opt .result{color:var(--muted)}
.effects{display:flex; flex-wrap:wrap; gap:6px}
.eff{font:500 11.5px/1 var(--mono); padding:5px 8px; border-radius:5px; background:var(--chip); white-space:nowrap}
.eff.up{color:var(--up)} .eff.down{color:var(--down)}
.eff.flag{color:var(--accent)} .eff.note{color:var(--brass); white-space:normal}
.kind{font:500 10.5px/1 var(--mono); letter-spacing:.06em; text-transform:uppercase; color:var(--faint)}
/* scenes */
.controls{display:flex; flex-wrap:wrap; gap:10px; align-items:end; margin:14px 0}
.controls label{display:flex; flex-direction:column; gap:4px; font:500 11px/1 var(--mono); letter-spacing:.06em; text-transform:uppercase; color:var(--muted)}
select,input[type=search]{font:14px var(--body); color:var(--ink); background:var(--surface); border:1px solid var(--rule); border-radius:6px; padding:7px 10px; min-width:0; max-width:100%}
input[type=search]{width:min(320px,100%)}
select:focus-visible,input:focus-visible,button:focus-visible{outline:2px solid var(--accent); outline-offset:1px}
.scenes{display:flex; flex-direction:column; gap:10px}
.scene{background:var(--surface); border:1px solid var(--rule); border-radius:8px; padding:12px 14px}
.scene .head{display:flex; flex-wrap:wrap; gap:6px 10px; align-items:baseline; margin-bottom:8px}
.scene .head .n{font:500 12px/1 var(--mono); color:var(--faint)}
.scene .head .t{font-variant:small-caps; letter-spacing:.02em}
.pair{display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:10px}
.pair>div{border-radius:6px; padding:8px 10px; background:var(--sunk)}
.pair .who{font:500 10.5px/1 var(--mono); letter-spacing:.06em; text-transform:uppercase; color:var(--muted); display:block; margin-bottom:5px}
.pair .claude{background:var(--accent-soft)}
.pair .over{color:var(--down); font:500 10.5px var(--mono)}
@media (max-width:640px){ .pair{grid-template-columns:1fr} details.sit>summary{grid-template-columns:1fr} .sit .meta{justify-content:flex-start} }
button.more{font:500 14px var(--body); color:var(--accent); background:var(--surface); border:1px solid var(--rule); border-radius:6px; padding:9px 16px; cursor:pointer; margin-top:12px}
.count{font:500 12px var(--mono); color:var(--muted)}
footer{margin-top:40px; padding-top:16px; border-top:1px solid var(--rule); color:var(--faint); font-size:12.5px}
@media (prefers-reduced-motion: reduce){ *{scroll-behavior:auto!important} }
</style>
<div class="wrap">
<header class="top">
<div class="eyebrow">story-to-pack · reference pass · 2026-09-17</div>
<h1>The Four Million, read by hand</h1>
<p class="lede">Every one of the 839 scenes in the v2 O. Henry corpus, described by Claude under the same rules the local models were given, then grouped into recurring situations and turned into draft events with options. It is a yardstick for the pipeline and for qwen3:14b, not a pack.</p>
<div class="facts" id="facts"></div>
</header>
<nav class="jump" aria-label="Sections">
<a href="#situations">Situations &amp; options</a>
<a href="#compare">Against the local pipeline</a>
<a href="#scenes">All 839 scenes</a>
<a href="#method">How this was made</a>
</nav>
<section id="situations">
<h2>Situations the corpus keeps returning to</h2>
<p>Twenty groups. Sixteen pass the pipeline's own candidate rule: at least five scenes, at least four stories, and no story over 40%. The other four are real but narrow, and are marked. Each group becomes one event (two for the benefactor, one per side) with three to five options. Effects name a state path, a direction and a rough size, never a number, as DESIGN.md requires.</p>
<div class="note" id="spine"></div>
<div class="sitlist" id="sitlist"></div>
</section>
<section id="compare">
<h2>Against the local pipeline</h2>
<h3 style="margin-top:8px">One-sentence predicaments, same rules, 839 scenes each</h3>
<p>The prompt asked for at most 20 words and no names, jobs, professions, places or businesses. Topic words are the pipeline's own <code>topic_words.py</code> list, which also counts relationship words.</p>
<div class="tablewrap"><table id="texttable"></table></div>
<h3 style="margin-top:22px">Do the 14B groups find these situations?</h3>
<p>These are the k-means groups on the qwen3:14b predicament embeddings at k = 60, compared with the hand grouping over the 475 scenes that carry a situation. An adjusted Rand index of 0 means chance agreement and 1 means identical groups.</p>
<div class="tablewrap"><table id="aritable"></table></div>
<p>At the review seed (24), each hand situation is spread across many 14B groups:</p>
<div class="tablewrap"><table id="spreadtable"></table></div>
<div class="note" id="caveat"></div>
</section>
<section id="scenes">
<h2>All 839 scenes, side by side</h2>
<p>Claude's summary and predicament next to the pipeline's (qwen2.5:3b summary, qwen3:14b predicament). Filter by situation or story, or search any of the four texts.</p>
<div class="controls">
<label for="f-sit">Situation<select id="f-sit"></select></label>
<label for="f-story">Story<select id="f-story"></select></label>
<label for="f-q">Search<input type="search" id="f-q" placeholder="word in any text"></label>
<span class="count" id="f-count"></span>
</div>
<div class="scenes" id="scenelist"></div>
<button class="more" id="more" type="button">Show 40 more</button>
</section>
<section id="method">
<h2>How this was made</h2>
<div id="methodbody"></div>
</section>
<footer id="foot"></footer>
</div>
<script id="data" type="application/json">__DATA__</script>
<script>
(function(){
const D = JSON.parse(document.getElementById('data').textContent);
const $ = s => document.querySelector(s);
const esc = s => String(s).replace(/[&<>"]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
const title = s => s.toLowerCase().replace(/\b\w/g, c => c.toUpperCase());
// facts
const F = D.facts;
$('#facts').innerHTML = F.map(f => `<div class="fact"><b>${esc(f[0])}</b><span>${esc(f[1])}</span></div>`).join('');
// spine
const S = D.situations;
$('#spine').innerHTML = `<p><b>Player role (assumed for the draft):</b> ${esc(S.player_role)}. <b>Stages:</b> ${Object.entries(S.stages).map(([k,v])=>`<code>${k}</code> ${esc(v)}`).join(' · ')}</p>
<p><b>Cast (roles, not people):</b> ${Object.entries(S.cast).map(([k,v])=>`<code>${k}</code> ${esc(v)}`).join(' · ')}. <b>Skills:</b> ${Object.entries(S.skills).map(([k,v])=>`<code>${k}</code> ${esc(v)}`).join(' · ')}. Relationship stats: ${S.relationship_stats.map(x=>`<code>${x}</code>`).join(' ')}.</p>`;
function effChip(e){
if (e.push) return `<span class="eff note">notable: “${esc(e.push)}”</span>`;
if (e.set) return `<span class="eff flag">${esc(e.path)} = true</span>`;
const up = e.dir === '+';
return `<span class="eff ${up?'up':'down'}">${esc(e.path)} ${up?'▲':'▼'} ${e.size[0].toUpperCase()}</span>`;
}
const sceneRef = cs => cs.length ? `scenes ${cs.join(', ')}` : '';
$('#sitlist').innerHTML = S.situations.map((s,i) => {
const ev0 = s.events[0];
const st = {}; s.members.forEach(m => { st[m.title] = (st[m.title]||0)+1; });
const storyList = Object.entries(st).sort((a,b)=>b[1]-a[1]).map(([t,n])=>`<i>${esc(t)}</i> ${n}`).join(' · ');
const gate = s.passes_pipeline_candidate_rule ? `<span class="pill ok">passes candidate rule</span>` :
`<span class="pill below">below the bar${s.stories<4?' · '+s.stories+' stories':''}${s.dominant_story_share>0.4?' · one story '+Math.round(s.dominant_story_share*100)+'%':''}</span>`;
const events = s.events.map(ev => `
<div class="event">
<h4>${esc(ev.title)}</h4>
<div class="ev-meta">Stages: ${ev.stages.map(x=>`<code>${x}</code>`).join(' ')} · Roles: ${ev.roles.map(x=>`<code>${x}</code>`).join(' ')}</div>
<p style="margin-top:6px">${esc(ev.sentence)}</p>
<ul class="descs">${ev.descriptions.map(d=>`<li>${esc(d.text)}<span class="src">from ${sceneRef(d.sources)}</span></li>`).join('')}</ul>
<div class="opts">${ev.options.map(o=>`
<div class="opt">
<div><span class="label">${esc(o.label)}</span> <span class="kind">${o.source.kind}${o.source.chunks.length?' · '+sceneRef(o.source.chunks):''}</span></div>
<div class="result">${esc(o.result)}</div>
<div class="effects">${o.effects.length? o.effects.map(effChip).join('') : '<span class="eff">no effect</span>'}</div>
</div>`).join('')}</div>
</div>`).join('');
return `<details class="sit" ${i<2?'open':''}>
<summary><h3>${esc(ev0.title)} <span class="src">${esc(s.id)}</span></h3>
<div class="meta"><span class="pill">${s.scenes} scenes</span><span class="pill">${s.stories} stories</span>${gate}</div>
<p class="sentence">${esc(ev0.sentence)}</p></summary>
<div class="sitbody"><p class="stories">${storyList}</p>${events}</div></details>`;
}).join('');
// comparison tables
const T = D.text;
$('#texttable').innerHTML = `<thead><tr><th>Source</th><th class="num">Median words</th><th class="num">Over 20 words</th><th class="num">With a topic word</th><th class="num">With a place word</th><th class="num">Capitalised mid-sentence</th><th>Commonest topic words</th></tr></thead><tbody>` +
T.map(r=>`<tr><td>${esc(r.name)}</td><td class="num">${r.median}</td><td class="num">${r.over}</td><td class="num">${(r.topic*100).toFixed(1)}%</td><td class="num">${(r.place*100).toFixed(1)}%</td><td class="num">${r.caps}</td><td>${esc(r.top)}</td></tr>`).join('') + `</tbody>`;
const A = D.ari;
$('#aritable').innerHTML = `<thead><tr><th>Seed</th><th class="num">ARI</th><th class="num">Candidate groups</th><th class="num">Median share: commonest situation</th><th class="num">Median share: no situation</th></tr></thead><tbody>` +
A.map(r=>`<tr><td>${r.seed}${r.seed===24?' (review)':''}</td><td class="num">${r.ari.toFixed(3)}</td><td class="num">${r.groups}</td><td class="num">${r.purity}%</td><td class="num">${r.none}%</td></tr>`).join('') + `</tbody>`;
$('#spreadtable').innerHTML = `<thead><tr><th>Situation</th><th class="num">Scenes</th><th class="num">14B groups it spans</th><th class="num">Largest share in one group</th></tr></thead><tbody>` +
D.spread.map(r=>`<tr><td><code>${esc(r[0])}</code></td><td class="num">${r[1]}</td><td class="num">${r[2]}</td><td class="num">${r[3]}%</td></tr>`).join('') + `</tbody>`;
$('#caveat').innerHTML = D.caveat;
// scenes
const SC = D.scenes, SITS = [...new Set(SC.map(s=>s.sit).filter(Boolean))].sort();
$('#f-sit').innerHTML = `<option value="">All</option><option value="-">No situation</option>` + SITS.map(s=>`<option>${esc(s)}</option>`).join('');
const titles = []; SC.forEach(s => { if (!titles.includes(s.title)) titles.push(s.title); });
$('#f-story').innerHTML = `<option value="">All ${titles.length} stories</option>` + titles.map(t=>`<option value="${esc(t)}">${esc(title(t))}</option>`).join('');
let shown = 40, rows = SC;
function words(s){ return s.split(/\s+/).filter(Boolean).length; }
function render(){
const list = rows.slice(0, shown);
$('#scenelist').innerHTML = list.map(s => `
<article class="scene">
<div class="head"><span class="n">scene ${s.chunk}</span><span class="t">${esc(title(s.title))}</span>${s.sit?`<span class="pill ok">${esc(s.sit)}</span>`:'<span class="pill">no situation</span>'}</div>
<div class="pair">
<div class="claude"><span class="who">Claude · predicament</span>${esc(s.cp)}</div>
<div><span class="who">qwen3:14b · predicament ${words(s.qp)>20?`<span class="over">${words(s.qp)} words</span>`:''}</span>${esc(s.qp)}</div>
<div class="claude"><span class="who">Claude · summary</span>${esc(s.cs)}</div>
<div><span class="who">qwen2.5:3b · summary</span>${esc(s.qs)}</div>
</div>
</article>`).join('');
$('#f-count').textContent = `${Math.min(shown, rows.length)} of ${rows.length} scenes`;
$('#more').hidden = shown >= rows.length;
}
function apply(){
const sit = $('#f-sit').value, st = $('#f-story').value, q = $('#f-q').value.trim().toLowerCase();
rows = SC.filter(s => (!sit || (sit==='-' ? !s.sit : s.sit===sit)) && (!st || s.title===st) &&
(!q || (s.cp+' '+s.cs+' '+s.qp+' '+s.qs).toLowerCase().includes(q)));
shown = 40; render();
}
['#f-sit','#f-story'].forEach(id => $(id).addEventListener('change', apply));
$('#f-q').addEventListener('input', apply);
$('#more').addEventListener('click', () => { shown += 40; render(); });
render();
$('#methodbody').innerHTML = D.method;
$('#foot').innerHTML = D.foot;
})();
</script>