Jesse.Markowitz/TheLadder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace clustering with a catalogue, and hand-write the references to judge it against

Browse Source 
The pipeline's embed-and-cluster step is dead, and this commit holds both the
evidence for that and the step proposed to replace it.

Predicaments. Scenes are re-described as "what the person is up against", with
no names, jobs or places, then embedded and clustered (redescribe.py,
topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading
both side by side. Two defects the pilot exposed are fixed: split.py missed
titles in quotes and a contents subtitle after a dash, so three stories had been
merged into their neighbours, and strip_names.py read New York place names as
people. The corrected corpus is probe/v2 (97 stories, 839 scenes);
carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from
20% to 13% at k=60, short of the pre-registered 10%.

Hand references. Three corpora were read scene by scene and written up by hand,
under the same prompt rules the local models get, as a baseline to judge them
against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's
Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of
the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a
readable page. No inference was used for any of them.

Catalogue. probe/catalogue maps every hand group in the three references onto 36
situation entries, with an answer key per corpus and one recurrence rule applied
to all three. classify.py assigns a scene one entry or none, leave-one-corpus-
out; score.py checks it against the key, with a self-test on random labels.

Why clustering is out: hand-written predicaments, embedded and clustered exactly
as the model's were, agree with the hand grouping at ARI 0.05 — no better than
the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even
shortlist: the hand label is the nearest entry 13% of the time and in the top 8
half the time.

The classification runs are not here. The dev and test runs are pre-registered
in probe/catalogue/README.md with the bar set beforehand, and are blocked on the
inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene
partial output in out/ is not a result.

Review page. The situation review is now a browser page rather than JSON edited
by hand (review_page.py, review_page_logic.cjs with Node tests, format schema
v2). It has never been rendered in a real browser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
This commit is contained in:
JesseMarkowitz
2026-09-20 17:19:42 -04:00
co-authored by Claude Opus 5
parent fa3769d0fe
commit e5617b86ba
135 changed files with 82027 additions and 1778 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/story-to-pack/probe/v2/claude/compare-claude-embeddings.log
+80
View File
@@ -0,0 +1,80 @@
== 2. Claude hand situations vs 14B k-means groups (k = 60) ==
Claude assigned 475 of 839 scenes to 20 situations; the rest carry no recurring situation.
seed 24: ARI on assigned scenes +0.049; 52 candidate groups: median share of a group that is its commonest Claude situation 17%, median share with no situation 43%
how each Claude situation is scattered across the 14B groups at seed 24:
pose 52 scenes over 24 groups; largest single group holds 15%
benefactor 48 scenes over 24 groups; largest single group holds 19%
station 47 scenes over 22 groups; largest single group holds 23%
taken_for_granted 47 scenes over 29 groups; largest single group holds 11%
newcomer 32 scenes over 15 groups; largest single group holds 19%
easy_money 30 scenes over 21 groups; largest single group holds 13%
sacrifice 25 scenes over 16 groups; largest single group holds 12%
rival 24 scenes over 15 groups; largest single group holds 21%
price 23 scenes over 15 groups; largest single group holds 22%
despised 20 scenes over 14 groups; largest single group holds 20%
appearances 19 scenes over 13 groups; largest single group holds 21%
loyalty 19 scenes over 10 groups; largest single group holds 26%
sign 17 scenes over 9 groups; largest single group holds 35%
honest_work 15 scenes over 10 groups; largest single group holds 20%
strings 15 scenes over 14 groups; largest single group holds 13%
unspoken 13 scenes over 11 groups; largest single group holds 15%
overlooked 9 scenes over 7 groups; largest single group holds 33%
homecoming 8 scenes over 6 groups; largest single group holds 38%
patron 8 scenes over 7 groups; largest single group holds 25%
habit 4 scenes over 3 groups; largest single group holds 50%
the 14B candidate groups, purest first (share = scenes carrying the commonest Claude situation):
65% of 17 station also unspoken 2, pose 1, strings 1
62% of 13 pose also appearances 1, easy_money 1
60% of 10 sign also taken_for_granted 1, overlooked 1
56% of 9 price also benefactor 1
45% of 11 rival also station 1, patron 1, overlooked 1
43% of 14 newcomer also taken_for_granted 3, honest_work 1, habit 1
43% of 7 loyalty also taken_for_granted 1
41% of 22 benefactor also despised 4, sacrifice 2, station 1
37% of 19 pose also taken_for_granted 2, easy_money 1, station 1
33% of 6 loyalty also rival 1
27% of 11 price also newcomer 1
27% of 15 rival also easy_money 2, pose 1, appearances 1
25% of 8 unspoken also taken_for_granted 1
25% of 8 station also taken_for_granted 2
25% of 8 sacrifice also strings 1, pose 1, benefactor 1
24% of 21 loyalty also benefactor 3, sacrifice 3, honest_work 1
21% of 19 station also benefactor 3, sacrifice 2, despised 2
20% of 25 newcomer also station 2, benefactor 2, homecoming 1
20% of 15 loyalty also benefactor 2, despised 2, strings 1
20% of 10 easy_money also sacrifice 2, strings 1, benefactor 1
20% of 5 sacrifice
20% of 5 appearances also easy_money 1, benefactor 1
19% of 21 easy_money also station 3, pose 2, appearances 2
19% of 27 taken_for_granted also pose 3, station 2, easy_money 2
18% of 11 strings also sacrifice 2, unspoken 1, appearances 1
18% of 11 benefactor also easy_money 1, despised 1, station 1
17% of 6 rival also price 1, station 1, loyalty 1
16% of 19 taken_for_granted also newcomer 2, station 1, easy_money 1
16% of 19 pose also newcomer 1, benefactor 1, sign 1
15% of 13 habit also taken_for_granted 2, sacrifice 1
15% of 13 easy_money also honest_work 2, strings 1, sign 1
14% of 14 station also benefactor 2, taken_for_granted 1, unspoken 1
14% of 14 price also benefactor 2, honest_work 1, newcomer 1
14% of 14 benefactor also pose 2, loyalty 1, sacrifice 1
13% of 15 taken_for_granted also pose 2, patron 2, easy_money 1
13% of 23 easy_money also benefactor 3, despised 2, homecoming 1
12% of 16 rival also strings 1, unspoken 1, station 1
12% of 17 sacrifice also unspoken 1, easy_money 1, rival 1
11% of 27 pose also benefactor 2, rival 2, appearances 2
11% of 18 station also benefactor 2, sacrifice 2, taken_for_granted 1
11% of 18 sign also benefactor 2, rival 1, easy_money 1
11% of 18 price also taken_for_granted 2, station 1, strings 1
11% of 9 strings also taken_for_granted 1, despised 1
11% of 9 price also pose 1, patron 1
11% of 37 station also newcomer 4, pose 4, overlooked 3
11% of 19 honest_work also appearances 2, newcomer 2, unspoken 1
10% of 20 newcomer also strings 1, benefactor 1, sign 1
10% of 42 appearances also station 3, homecoming 3, sign 3
10% of 21 pose also benefactor 2, newcomer 2, strings 1
9% of 11 honest_work also price 1, easy_money 1, loyalty 1
7% of 14 homecoming
0% of 5 -
seed 11: ARI on assigned scenes +0.051; 55 candidate groups: median share of a group that is its commonest Claude situation 20%, median share with no situation 42%
seed 12: ARI on assigned scenes +0.046; 52 candidate groups: median share of a group that is its commonest Claude situation 20%, median share with no situation 42%
seed 13: ARI on assigned scenes +0.049; 51 candidate groups: median share of a group that is its commonest Claude situation 17%, median share with no situation 42%