Replace clustering with a catalogue, and hand-write the references to judge it against
The pipeline's embed-and-cluster step is dead, and this commit holds both the evidence for that and the step proposed to replace it. Predicaments. Scenes are re-described as "what the person is up against", with no names, jobs or places, then embedded and clustered (redescribe.py, topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading both side by side. Two defects the pilot exposed are fixed: split.py missed titles in quotes and a contents subtitle after a dash, so three stories had been merged into their neighbours, and strip_names.py read New York place names as people. The corrected corpus is probe/v2 (97 stories, 839 scenes); carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from 20% to 13% at k=60, short of the pre-registered 10%. Hand references. Three corpora were read scene by scene and written up by hand, under the same prompt rules the local models get, as a baseline to judge them against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a readable page. No inference was used for any of them. Catalogue. probe/catalogue maps every hand group in the three references onto 36 situation entries, with an answer key per corpus and one recurrence rule applied to all three. classify.py assigns a scene one entry or none, leave-one-corpus- out; score.py checks it against the key, with a self-test on random labels. Why clustering is out: hand-written predicaments, embedded and clustered exactly as the model's were, agree with the hand grouping at ARI 0.05 — no better than the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even shortlist: the hand label is the nearest entry 13% of the time and in the top 8 half the time. The classification runs are not here. The dev and test runs are pre-registered in probe/catalogue/README.md with the bar set beforehand, and are blocked on the inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene partial output in out/ is not a result. Review page. The situation review is now a browser page rather than JSON edited by hand (review_page.py, review_page_logic.cjs with Node tests, format schema v2). It has never been rendered in a real browser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
This commit is contained in:
co-authored by
Claude Opus 5
parent
fa3769d0fe
commit
e5617b86ba
@@ -0,0 +1,15 @@
|
||||
[
|
||||
{"chunk":199,"story":7,"summary":"A newly engaged man drops in on the woman he has long kept as an intimate friend, meaning to talk over her meeting with his fiancée.","predicament":"Someone who has moved on to a new commitment returns to an old intimate as if nothing had changed."},
|
||||
{"chunk":200,"story":7,"summary":"He prides himself on an art of evading declared feeling, having once been burdened by an avowed love.","predicament":"Someone who once regretted committing themselves now takes pleasure in never committing to anyone."},
|
||||
{"chunk":201,"story":7,"summary":"He trained his friend over years to hide her feelings and follow his lead in every evasion.","predicament":"Someone has shaped another person to fit their convenience and admires the result."},
|
||||
{"chunk":202,"story":7,"summary":"She received his fiancée perfectly; he regrets having announced the engagement by letter, as if he had doubted her.","predicament":"Someone realises that a cautious move of theirs betrayed distrust of a person who deserved better."},
|
||||
{"chunk":203,"story":7,"summary":"Missing his candid fiancée, he calls to talk about her and delights in the fresh feelings she stirs in him.","predicament":"Someone newly in love seeks out an old confidant to share it, without thinking what it costs them."},
|
||||
{"chunk":204,"story":7,"summary":"His friend is startled to see him and speaks of this as a last visit; he insists nothing has changed.","predicament":"Someone expects an old arrangement to continue and is told it is ending."},
|
||||
{"chunk":205,"story":7,"summary":"She reveals that his fiancée came to see her a second time that morning, which he did not know.","predicament":"Someone learns that the person they are about to marry went behind their back to an old intimate."},
|
||||
{"chunk":206,"story":7,"summary":"She pities him and hints that a letter may await him; the fiancée came to find out whether anything had happened between them.","predicament":"Someone discovers their fiancée has been asking about their past with another person."},
|
||||
{"chunk":207,"story":7,"summary":"He protests that a man's past is his own; she says the fiancée objects even to a mere friendship, and came to see whether he was really free.","predicament":"Someone who thought their past was their own business finds it standing in the way of their future."},
|
||||
{"chunk":208,"story":7,"summary":"Told there had never been an affair, the fiancée was more disturbed, not less: the offence was that nothing had happened.","predicament":"Someone who thought themselves innocent learns the harm lay in what they withheld."},
|
||||
{"chunk":209,"story":7,"summary":"The friend confesses she loved him all along; the fiancée saw that he used her and cast her aside, and despised them both.","predicament":"Someone is made to see, through an outsider's eyes, how they have used another person's devotion."},
|
||||
{"chunk":210,"story":7,"summary":"The fiancée wants a man capable of love, not reservations; the friend offers to say she lied, and let him claim a past with her.","predicament":"Someone who has been used offers to sacrifice their own name to save the person who used them."},
|
||||
{"chunk":211,"story":7,"summary":"Moved, he refuses her sacrifice, saying it would only harm his fiancée; she tells him the fiancée has surely recalled her letter.","predicament":"Someone offered an easy escape at another's expense refuses it, and learns they may not need one."}
|
||||
]
|
||||
Reference in New Issue
Block a user