Jesse.Markowitz/TheLadder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace clustering with a catalogue, and hand-write the references to judge it against

Browse Source 
The pipeline's embed-and-cluster step is dead, and this commit holds both the
evidence for that and the step proposed to replace it.

Predicaments. Scenes are re-described as "what the person is up against", with
no names, jobs or places, then embedded and clustered (redescribe.py,
topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading
both side by side. Two defects the pilot exposed are fixed: split.py missed
titles in quotes and a contents subtitle after a dash, so three stories had been
merged into their neighbours, and strip_names.py read New York place names as
people. The corrected corpus is probe/v2 (97 stories, 839 scenes);
carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from
20% to 13% at k=60, short of the pre-registered 10%.

Hand references. Three corpora were read scene by scene and written up by hand,
under the same prompt rules the local models get, as a baseline to judge them
against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's
Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of
the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a
readable page. No inference was used for any of them.

Catalogue. probe/catalogue maps every hand group in the three references onto 36
situation entries, with an answer key per corpus and one recurrence rule applied
to all three. classify.py assigns a scene one entry or none, leave-one-corpus-
out; score.py checks it against the key, with a self-test on random labels.

Why clustering is out: hand-written predicaments, embedded and clustered exactly
as the model's were, agree with the hand grouping at ARI 0.05 — no better than
the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even
shortlist: the hand label is the nearest entry 13% of the time and in the top 8
half the time.

The classification runs are not here. The dev and test runs are pre-registered
in probe/catalogue/README.md with the bar set beforehand, and are blocked on the
inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene
partial output in out/ is not a result.

Review page. The situation review is now a browser page rather than JSON edited
by hand (review_page.py, review_page_logic.cjs with Node tests, format schema
v2). It has never been rendered in a real browser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
This commit is contained in:
JesseMarkowitz
2026-09-20 17:19:42 -04:00
co-authored by Claude Opus 5
parent fa3769d0fe
commit e5617b86ba
135 changed files with 82027 additions and 1778 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/story-to-pack/probe/wharton/claude/groups.json
+262
View File
@@ -0,0 +1,262 @@
{
"method": "Read all 262 predicament sentences and grouped them by hand, by what the main person is up against and wants. Each scene is in at most one group. A group must span >= 3 of the 10 stories with no story above 50% of its members (O. Henry used >= 4 stories and <= 40% over 97 stories; this corpus has 10). Scenes left out are one-offs or narration. 'tightness' is my own label: 2 = one predicament, most members fit; 1 = broad, a shared shape that varies.",
"groups": [
{
"id": "tainted_money",
"name": "Money on terms you despise",
"predicament": "Someone is offered money or success on condition they go along with something they think is wrong, and tells themselves it will cost nothing that matters.",
"tightness": 2,
"members": [
12,
14,
22,
23,
27,
28,
53,
79,
87,
230,
232,
233
]
},
{
"id": "strings_attached",
"name": "A way out, with strings attached",
"predicament": "Someone who needs help is offered it, but only on terms that will cost them something they value.",
"tightness": 1,
"members": [
26,
89,
215,
256,
257
]
},
{
"id": "old_attachment",
"name": "An old attachment comes back",
"predicament": "Someone's present commitment is disturbed by a former partner or intimate who still has a claim, or a hold, on them or on the person they love.",
"tightness": 2,
"members": [
30,
33,
34,
39,
52,
55,
168,
169,
170,
199,
204,
207
]
},
{
"id": "what_they_hid",
"name": "Finding out what they hid",
"predicament": "Someone discovers that a person close to them has been hiding something, and must decide what it changes.",
"tightness": 2,
"members": [
44,
46,
120,
122,
159,
193,
205,
206,
259,
260
]
},
{
"id": "keep_their_secret",
"name": "Keeping someone else's secret",
"predicament": "Someone is drawn into another person's secret and must decide whether to lie, stay silent, or give it up when pressed.",
"tightness": 2,
"members": [
105,
107,
197,
251,
252
]
},
{
"id": "exposed",
"name": "About to be exposed in public",
"predicament": "Someone's work or private arrangement is about to be judged in front of the very people they most feared would see it.",
"tightness": 1,
"members": [
10,
17,
18,
58,
70,
73,
74,
84,
85,
145,
147,
244
]
},
{
"id": "civil_to_them",
"name": "Civil to someone you would rather avoid",
"predicament": "Someone must deal politely, even helpfully, with a person they resent or fear, because their position leaves them no choice.",
"tightness": 2,
"members": [
40,
41,
43,
49,
64,
96,
98,
217
]
},
{
"id": "plead_their_case",
"name": "Pleading someone else's case",
"predicament": "Someone is asked to carry another person's claim or warning to a third party, knowing it may cost them with one side or the other.",
"tightness": 2,
"members": [
47,
141,
142,
218,
219,
220
]
},
{
"id": "own_words_back",
"name": "Your own words turned against you",
"predicament": "Someone is held to a principle, promise or code they once used for their own ends, now that it works against them.",
"tightness": 2,
"members": [
160,
161,
164,
222,
253,
254
]
},
{
"id": "give_it_up",
"name": "Giving something up for someone else",
"predicament": "Someone can protect or free another person only by giving up something of their own, and the other may never know.",
"tightness": 2,
"members": [
125,
198,
210,
211,
221,
229,
236,
258
]
},
{
"id": "keep_the_peace",
"name": "Giving ground to keep the peace",
"predicament": "Someone gives way, again and again, to a spouse or household, because refusing would cost more than it is worth.",
"tightness": 2,
"members": [
21,
51,
57,
69,
130,
131,
136,
235
]
},
{
"id": "unseen_at_home",
"name": "Unseen at home",
"predicament": "Someone lives beside a spouse who does not really see them, whether through neglect, blind approval, or plain coarseness.",
"tightness": 2,
"members": [
2,
71,
72,
97,
123,
126,
127
]
},
{
"id": "new_and_warned",
"name": "New, and warned",
"predicament": "Someone new to a place or a household is warned, or senses, that something is wrong, and must decide whom to believe.",
"tightness": 1,
"members": [
31,
32,
90,
91,
95,
101,
241,
242
]
},
{
"id": "the_rival",
"name": "The rival",
"predicament": "Someone who has lost, or fears losing, the person they want to a rival is tempted to strike at the rival.",
"tightness": 2,
"members": [
37,
63,
153,
165,
186,
188,
189,
190
]
},
{
"id": "owning_up",
"name": "Owning up",
"predicament": "Someone who once wronged another faces them, or a stand-in for them, and must decide how much to admit.",
"tightness": 2,
"members": [
171,
172,
173,
176,
202,
209,
227,
228
]
},
{
"id": "waiting_on_them",
"name": "Waiting on someone else's decision",
"predicament": "Someone's hopes hang on a decision entirely in another's hands, and nothing they do seems to hurry it.",
"tightness": 1,
"members": [
116,
139,
140,
143,
212,
226
]
}
]
}