JesseMarkowitzandClaude Opus 5
e5617b86ba
The pipeline's embed-and-cluster step is dead, and this commit holds both the evidence for that and the step proposed to replace it. Predicaments. Scenes are re-described as "what the person is up against", with no names, jobs or places, then embedded and clustered (redescribe.py, topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading both side by side. Two defects the pilot exposed are fixed: split.py missed titles in quotes and a contents subtitle after a dash, so three stories had been merged into their neighbours, and strip_names.py read New York place names as people. The corrected corpus is probe/v2 (97 stories, 839 scenes); carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from 20% to 13% at k=60, short of the pre-registered 10%. Hand references. Three corpora were read scene by scene and written up by hand, under the same prompt rules the local models get, as a baseline to judge them against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a readable page. No inference was used for any of them. Catalogue. probe/catalogue maps every hand group in the three references onto 36 situation entries, with an answer key per corpus and one recurrence rule applied to all three. classify.py assigns a scene one entry or none, leave-one-corpus- out; score.py checks it against the key, with a self-test on random labels. Why clustering is out: hand-written predicaments, embedded and clustered exactly as the model's were, agree with the hand grouping at ARI 0.05 — no better than the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even shortlist: the hand label is the nearest entry 13% of the time and in the top 8 half the time. The classification runs are not here. The dev and test runs are pre-registered in probe/catalogue/README.md with the bar set beforehand, and are blocked on the inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene partial output in out/ is not a result. Review page. The situation review is now a browser page rather than JSON edited by hand (review_page.py, review_page_logic.cjs with Node tests, format schema v2). It has never been rendered in a real browser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
71 lines
6.3 KiB
Python
71 lines
6.3 KiB
Python
"""Build claude-reference.html from the Claude pass and the pipeline's files. No inference.
|
||
cd probe/v2 && python3 claude/report_page.py
|
||
"""
|
||
import json, pathlib, re, statistics, collections, sys
|
||
HERE = pathlib.Path(__file__).resolve().parent
|
||
sys.path.insert(0, str(HERE)); sys.path.insert(0, str(HERE.parent.parent))
|
||
from assign import ASSIGN
|
||
from topic_words import TOPIC_WORDS, PLACE_WORDS
|
||
|
||
chunks = json.load(open('chunks.json'))
|
||
n = len(chunks)
|
||
cp = {x['chunk']: x['summary'] for x in json.load(open(HERE / 'predicaments-claude.json'))['summaries']}
|
||
cs = {x['chunk']: x['summary'] for x in json.load(open(HERE / 'summaries-claude.json'))['summaries']}
|
||
qp = {x['chunk']: x['summary'] for x in json.load(open('predicaments_clean.json'))['summaries']}
|
||
qpraw = {x['chunk']: x.get('raw', x['summary']) for x in json.load(open('predicaments.json'))['summaries']}
|
||
qs = {x['chunk']: x['summary'] for x in json.load(open('summaries_clean.json'))['summaries']}
|
||
sit = json.load(open(HERE / 'situations-claude.json'))
|
||
|
||
def w(s): return re.findall(r"[a-zé']+", s.lower())
|
||
def caps(s): return [t for t in re.findall(r"[A-Za-z']+", s)[1:] if t[0].isupper() and t != 'I']
|
||
text = []
|
||
for name, d in (('Claude (claude-opus-5)', cp), ('qwen3:14b, as generated', qpraw), ('qwen3:14b after strip_names.py', qp)):
|
||
tw = [set(w(d[i])) & TOPIC_WORDS for i in range(n)]
|
||
text.append(dict(name=name, median=statistics.median(len(d[i].split()) for i in range(n)),
|
||
over=sum(len(d[i].split()) > 20 for i in range(n)), topic=sum(bool(x) for x in tw) / n,
|
||
place=sum(bool(set(w(d[i])) & PLACE_WORDS) for i in range(n)) / n, caps=sum(bool(caps(d[i])) for i in range(n)),
|
||
top=', '.join(f'{k} {v}' for k, v in collections.Counter(x for s in tw for x in s).most_common(8))))
|
||
|
||
# grouping numbers come from compare.log so the page never disagrees with the log
|
||
log = (HERE / 'compare.log').read_text()
|
||
ari = [dict(seed=int(m[0]), ari=float(m[1]), groups=int(m[2]), purity=int(m[3]), none=int(m[4])) for m in
|
||
re.findall(r"seed\s+(\d+): ARI on assigned scenes ([+-][\d.]+); (\d+) candidate groups: median share of a group that is its commonest Claude situation (\d+)%, median share with no situation (\d+)%", log)]
|
||
spread = [(m[0], int(m[1]), int(m[2]), int(m[3])) for m in re.findall(r"^\s+(\w+)\s+(\d+) scenes over\s+(\d+) groups; largest single group holds\s+(\d+)%", log, re.M)]
|
||
assert len(ari) == 4 and len(spread) == 20, (len(ari), len(spread))
|
||
|
||
scenes = [dict(chunk=i, title=chunks[i]['title'], cp=cp[i], cs=cs[i], qp=qp[i], qs=qs[i], sit=ASSIGN.get(i, '')) for i in range(n)]
|
||
n_sit = len(sit['situations']); passing = sum(s['passes_pipeline_candidate_rule'] for s in sit['situations'])
|
||
events = sum(len(s['events']) for s in sit['situations']); opts = sum(len(e['options']) for s in sit['situations'] for e in s['events'])
|
||
facts = [(f'{n}', 'scenes described, 97 stories'), (f'{len(ASSIGN)}', f'scenes carrying a recurring situation ({len(ASSIGN)/n:.0%})'),
|
||
(f'{passing} of {n_sit}', 'situations pass the pipeline’s candidate rule'), (f'{events} · {opts}', 'draft events · options'),
|
||
(f"{ari[0]['ari']:.2f}", 'agreement with the 14B groups (ARI, seed 24)')]
|
||
|
||
caveat = ("<p><b>What this does and does not show.</b> The hand grouping is one careful reader’s judgment, not ground truth. "
|
||
"It is, however, the kind of reading the pipeline is trying to automate, and the 14B groups barely track it: a median 14B candidate group is "
|
||
f"{ari[0]['purity']}% one hand situation and {ari[0]['none']}% scenes with no recurring situation at all. This comparison cannot say whether the fault is the "
|
||
"14B predicament text or the embed-and-cluster step. The way to separate them is to embed <i>these</i> 839 predicaments with nomic-embed-text and run the same "
|
||
"k-means. That needs the inference host, so it has not been run.</p>"
|
||
"<p><b>A caveat on my own text:</b> 88% of my predicaments begin “Someone”. That shared opener is harmless to a reader, but "
|
||
"DESIGN.md already flags “someone” as a token that could pull embeddings together, so it may make my file look more alike to an embedder than it reads.</p>")
|
||
|
||
method = ("<p><b>Scenes.</b> <code>probe/v2/chunks.json</code>, 839 scenes from 97 stories, read in order, one story at a time, in 19 batches. Each scene got a "
|
||
"summary under the qwen2.5:3b summary prompt (one sentence, at most 25 words, roles not names, no places) and a predicament under the qwen3:14b redescribe "
|
||
"prompt (one sentence, at most 20 words, no names, jobs, places, businesses or story-specific objects). I kept names out as I wrote; nothing was "
|
||
"stripped afterwards. Every record was checked for length and story id.</p>"
|
||
"<p><b>Grouping.</b> By hand, from the predicaments and summaries together (<code>claude/assign.py</code>). Each scene gets at most one situation. "
|
||
"364 scenes carry none: essays, set-ups and one-off turns. No embeddings and no clustering were used.</p>"
|
||
"<p><b>Events.</b> One event per situation, with the benefactor split in two, because the side you stand on changes every option. The options are "
|
||
"tagged <code>scene</code> (taken from a named scene), <code>archetype</code> (the fixed moves DESIGN.md calls for, such as “say nothing”) or "
|
||
"<code>gap</code> (written to fill a hole). Descriptions are generic second person, with the scenes they came from. Sizes are advisory priors for the "
|
||
"numeric search. <code>claude/situations_src.py</code> checks option counts, paths, capitalised names and duplicate effect sets.</p>"
|
||
"<p><b>Comparison.</b> <code>claude/compare.py</code> uses only files already on disk. No model was called for any of it.</p>"
|
||
"<p><b>Files</b>, all in <code>probe/v2/claude/</code>: <code>summaries-claude.json</code> and <code>predicaments-claude.json</code> "
|
||
"(same shape as the pipeline's), <code>assign.py</code>, <code>situations-claude.json</code>, <code>compare.log</code>, this page.</p>")
|
||
foot = "Corpus: O. Henry, The Four Million cycle (Project Gutenberg, public domain). Built by <code>claude/report_page.py</code>."
|
||
|
||
data = dict(facts=facts, situations=sit, text=text, ari=ari, spread=spread, caveat=caveat, scenes=scenes, method=method, foot=foot)
|
||
blob = json.dumps(data, ensure_ascii=False).replace('</', '<\\/')
|
||
html = (HERE / 'report_template.html').read_text().replace('__DATA__', blob)
|
||
(HERE / 'claude-reference.html').write_text(html, encoding='utf-8')
|
||
print('wrote claude/claude-reference.html', len(html) // 1024, 'KB')
|