Jesse.Markowitz/TheLadder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace clustering with a catalogue, and hand-write the references to judge it against

Browse Source 
The pipeline's embed-and-cluster step is dead, and this commit holds both the
evidence for that and the step proposed to replace it.

Predicaments. Scenes are re-described as "what the person is up against", with
no names, jobs or places, then embedded and clustered (redescribe.py,
topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading
both side by side. Two defects the pilot exposed are fixed: split.py missed
titles in quotes and a contents subtitle after a dash, so three stories had been
merged into their neighbours, and strip_names.py read New York place names as
people. The corrected corpus is probe/v2 (97 stories, 839 scenes);
carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from
20% to 13% at k=60, short of the pre-registered 10%.

Hand references. Three corpora were read scene by scene and written up by hand,
under the same prompt rules the local models get, as a baseline to judge them
against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's
Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of
the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a
readable page. No inference was used for any of them.

Catalogue. probe/catalogue maps every hand group in the three references onto 36
situation entries, with an answer key per corpus and one recurrence rule applied
to all three. classify.py assigns a scene one entry or none, leave-one-corpus-
out; score.py checks it against the key, with a self-test on random labels.

Why clustering is out: hand-written predicaments, embedded and clustered exactly
as the model's were, agree with the hand grouping at ARI 0.05 — no better than
the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even
shortlist: the hand label is the nearest entry 13% of the time and in the top 8
half the time.

The classification runs are not here. The dev and test runs are pre-registered
in probe/catalogue/README.md with the bar set beforehand, and are blocked on the
inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene
partial output in out/ is not a result.

Review page. The situation review is now a browser page rather than JSON edited
by hand (review_page.py, review_page_logic.cjs with Node tests, format schema
v2). It has never been rendered in a real browser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
This commit is contained in:
JesseMarkowitz
2026-09-20 17:19:42 -04:00
co-authored by Claude Opus 5
parent fa3769d0fe
commit e5617b86ba
135 changed files with 82027 additions and 1778 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/story-to-pack/probe/v2/claude/compare.log
+76
View File
@@ -0,0 +1,76 @@
== 1. Predicament text (839 scenes each) ==
claude median words 14 over 20 words 0 with a topic word 8.3% with a place word 3.6% capitalised mid-sentence 0 starts "Someone" 88%
commonest topic words: room 13, rival 11, partner 10, suitor 5, street 5, judge 4, station 3, house 3, employer 3, court 2, neighbour 2, stage 2
qwen3:14b raw median words 24 over 20 words 709 with a topic word 23.0% with a place word 10.1% capitalised mid-sentence 110 starts "Someone" 0%
commonest topic words: wife 16, lover 10, husband 9, street 9, restaurant 9, room 8, writer 7, stage 6, dinner 6, cop 5, park 5, hotel 5
qwen3:14b after strip_names median words 24 over 20 words 707 with a topic word 23.0% with a place word 10.1% capitalised mid-sentence 35 starts "Someone" 20%
commonest topic words: wife 16, lover 10, husband 9, street 9, restaurant 9, room 8, writer 7, stage 6, dinner 6, cop 5, park 5, hotel 5
== 2. Claude hand situations vs 14B k-means groups (k = 60) ==
Claude assigned 475 of 839 scenes to 20 situations; the rest carry no recurring situation.
seed 24: ARI on assigned scenes +0.075; 40 candidate groups: median share of a group that is its commonest Claude situation 20%, median share with no situation 41%
how each Claude situation is scattered across the 14B groups at seed 24:
pose 52 scenes over 23 groups; largest single group holds 15%
benefactor 48 scenes over 21 groups; largest single group holds 17%
station 47 scenes over 19 groups; largest single group holds 30%
taken_for_granted 47 scenes over 25 groups; largest single group holds 30%
newcomer 32 scenes over 16 groups; largest single group holds 38%
easy_money 30 scenes over 15 groups; largest single group holds 17%
sacrifice 25 scenes over 16 groups; largest single group holds 24%
rival 24 scenes over 12 groups; largest single group holds 33%
price 23 scenes over 10 groups; largest single group holds 22%
despised 20 scenes over 18 groups; largest single group holds 10%
appearances 19 scenes over 10 groups; largest single group holds 21%
loyalty 19 scenes over 9 groups; largest single group holds 32%
sign 17 scenes over 9 groups; largest single group holds 24%
honest_work 15 scenes over 8 groups; largest single group holds 40%
strings 15 scenes over 11 groups; largest single group holds 13%
unspoken 13 scenes over 11 groups; largest single group holds 15%
overlooked 9 scenes over 8 groups; largest single group holds 22%
homecoming 8 scenes over 6 groups; largest single group holds 38%
patron 8 scenes over 7 groups; largest single group holds 25%
habit 4 scenes over 4 groups; largest single group holds 25%
the 14B candidate groups, purest first (share = scenes carrying the commonest Claude situation):
61% of 23 taken_for_granted also unspoken 1, sacrifice 1
50% of 6 price also station 1
40% of 15 honest_work also sacrifice 2, easy_money 1, unspoken 1
38% of 32 newcomer also benefactor 2, pose 1, homecoming 1
33% of 15 easy_money also benefactor 4, despised 1
33% of 12 appearances also easy_money 2, newcomer 1, pose 1
33% of 9 easy_money also appearances 2, newcomer 1, honest_work 1
27% of 15 easy_money also loyalty 1, despised 1, station 1
25% of 16 pose also price 1, station 1, taken_for_granted 1
23% of 60 station also pose 8, rival 8, sacrifice 2
23% of 13 station also benefactor 2, homecoming 1, newcomer 1
22% of 9 pose also taken_for_granted 1
22% of 9 overlooked also patron 2, honest_work 1, rival 1
21% of 28 loyalty also pose 3, homecoming 3, station 2
21% of 19 pose also strings 2, station 2, easy_money 2
21% of 29 benefactor also strings 2, loyalty 2, sacrifice 1
20% of 35 pose also price 4, appearances 3, despised 2
20% of 25 easy_money also loyalty 3, newcomer 2, strings 1
20% of 5 overlooked also rival 1
20% of 5 overlooked also taken_for_granted 1, station 1, patron 1
20% of 5 benefactor also newcomer 1, despised 1
19% of 42 benefactor also station 3, sign 3, pose 3
18% of 22 sign also station 3, price 2, habit 1
18% of 17 appearances also rival 2, strings 1, pose 1
17% of 6 pose also overlooked 1
16% of 19 pose also station 1, strings 1, taken_for_granted 1
16% of 32 benefactor also strings 2, loyalty 2, appearances 1
15% of 13 taken_for_granted also station 2, newcomer 2, benefactor 1
15% of 34 price also station 4, pose 3, rival 3
14% of 7 taken_for_granted
13% of 15 strings also benefactor 2, unspoken 1, easy_money 1
13% of 15 rival also despised 2, sign 1, pose 1
12% of 8 strings also taken_for_granted 1, benefactor 1, patron 1
12% of 25 sacrifice also price 3, newcomer 2, station 2
12% of 34 benefactor also taken_for_granted 3, newcomer 3, honest_work 2
11% of 9 despised also benefactor 1, newcomer 1, price 1
10% of 21 loyalty also benefactor 1, easy_money 1, honest_work 1
9% of 11 sign also easy_money 1
8% of 13 patron also easy_money 1, appearances 1, pose 1
6% of 31 station also sign 2, honest_work 1, appearances 1
seed 11: ARI on assigned scenes +0.061; 42 candidate groups: median share of a group that is its commonest Claude situation 20%, median share with no situation 41%
seed 12: ARI on assigned scenes +0.071; 45 candidate groups: median share of a group that is its commonest Claude situation 20%, median share with no situation 41%
seed 13: ARI on assigned scenes +0.060; 42 candidate groups: median share of a group that is its commonest Claude situation 19%, median share with no situation 40%