JesseMarkowitzandClaude Opus 5
e5617b86ba
The pipeline's embed-and-cluster step is dead, and this commit holds both the evidence for that and the step proposed to replace it. Predicaments. Scenes are re-described as "what the person is up against", with no names, jobs or places, then embedded and clustered (redescribe.py, topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading both side by side. Two defects the pilot exposed are fixed: split.py missed titles in quotes and a contents subtitle after a dash, so three stories had been merged into their neighbours, and strip_names.py read New York place names as people. The corrected corpus is probe/v2 (97 stories, 839 scenes); carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from 20% to 13% at k=60, short of the pre-registered 10%. Hand references. Three corpora were read scene by scene and written up by hand, under the same prompt rules the local models get, as a baseline to judge them against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a readable page. No inference was used for any of them. Catalogue. probe/catalogue maps every hand group in the three references onto 36 situation entries, with an answer key per corpus and one recurrence rule applied to all three. classify.py assigns a scene one entry or none, leave-one-corpus- out; score.py checks it against the key, with a self-test on random labels. Why clustering is out: hand-written predicaments, embedded and clustered exactly as the model's were, agree with the hand grouping at ARI 0.05 — no better than the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even shortlist: the hand label is the nearest entry 13% of the time and in the top 8 half the time. The classification runs are not here. The dev and test runs are pre-registered in probe/catalogue/README.md with the bar set beforehand, and are blocked on the inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene partial output in out/ is not a result. Review page. The situation review is now a browser page rather than JSON edited by hand (review_page.py, review_page_logic.cjs with Node tests, format schema v2). It has never been rendered in a real browser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
246 lines
10 KiB
JSON
246 lines
10 KiB
JSON
{
|
|
"model": "qwen2.5:3b-instruct",
|
|
"prompt": "You describe the predicament of the main person in a passage of fiction, in ONE sentence.\n\nA predicament is what that person is up against and what they want out of it: the kind of trouble or opportunity that could happen to anyone, in any line of work, in any town.\n\nRules:\n- Exactly one sentence, at most 20 words.\n- Leave out everything that ties it to this story: no names, no jobs or professions, no places, no businesses, and no objects that only belong to this story.\n- Say what the person is up against, and what they want.\n- Do not describe the writing, the mood, or who is telling the story.\n- Output only the sentence.",
|
|
"summaries": [
|
|
{
|
|
"chunk": 0,
|
|
"story": 0,
|
|
"summary": "Finding the voice of a big city, I sought in vain among school chants.",
|
|
"model_output": "Finding the voice of a big city, I sought in vain among school chants."
|
|
},
|
|
{
|
|
"chunk": 1,
|
|
"story": 0,
|
|
"summary": "I want to know the city's voice, but it echoes like Philadelphia.",
|
|
"model_output": "I want to know the city's voice, but it echoes like Philadelphia."
|
|
},
|
|
{
|
|
"chunk": 2,
|
|
"story": 0,
|
|
"summary": "I must find the unique voice of New York, proving it doesn't treat me like a cigar.",
|
|
"model_output": "I must find the unique voice of New York, proving it doesn't treat me like a cigar."
|
|
},
|
|
{
|
|
"chunk": 3,
|
|
"story": 0,
|
|
"summary": "I want to know the city's cry, but the cop only follows orders.",
|
|
"model_output": "I want to know the city's cry, but the cop only follows orders."
|
|
},
|
|
{
|
|
"chunk": 4,
|
|
"story": 0,
|
|
"summary": "Bill, find the essence of New York's chaos and harmony.",
|
|
"model_output": "Bill, find the essence of New York's chaos and harmony."
|
|
},
|
|
{
|
|
"chunk": 5,
|
|
"story": 0,
|
|
"summary": "The poet seeks a hit with his poem, fearing rejection.",
|
|
"model_output": "The poet seeks a hit with his poem, fearing rejection."
|
|
},
|
|
{
|
|
"chunk": 6,
|
|
"story": 0,
|
|
"summary": "Aurelia waits alone, wanting her companion to speak.",
|
|
"model_output": "Aurelia waits alone, wanting her companion to speak."
|
|
},
|
|
{
|
|
"chunk": 7,
|
|
"story": 1,
|
|
"summary": "John Hopkins finds deepest life joys in a hidden quarter, not wealth.",
|
|
"model_output": "John Hopkins finds deepest life joys in a hidden quarter, not wealth."
|
|
},
|
|
{
|
|
"chunk": 8,
|
|
"story": 1,
|
|
"summary": "Mrs. Hopkins struggles to balance her desires and fears in the fast-paced city.",
|
|
"model_output": "Mrs. Hopkins struggles to balance her desires and fears in the fast-paced city."
|
|
},
|
|
{
|
|
"chunk": 9,
|
|
"story": 1,
|
|
"summary": "John Hopkins sought to inject conversation into his tasteless existence by buying a cigar.",
|
|
"model_output": "John Hopkins sought to inject conversation into his tasteless existence by buying a cigar."
|
|
},
|
|
{
|
|
"chunk": 10,
|
|
"story": 1,
|
|
"summary": "Hopkins buys spinach, finds no cash, challenges tobacconist Freshmayer.",
|
|
"model_output": "Hopkins buys spinach, finds no cash, challenges tobacconist Freshmayer."
|
|
},
|
|
{
|
|
"chunk": 11,
|
|
"story": 1,
|
|
"summary": "John Hopkins sought revenge, but found himself pursued by law enforcers and a mysterious chauffeur in a racing car.",
|
|
"model_output": "John Hopkins sought revenge, but found himself pursued by law enforcers and a mysterious chauffeur in a racing car."
|
|
},
|
|
{
|
|
"chunk": 12,
|
|
"story": 1,
|
|
"summary": "Hopkins wants Milady's assistance against unseen attackers, while Milady seeks a brave companion.",
|
|
"model_output": "Hopkins wants Milady's assistance against unseen attackers, while Milady seeks a brave companion."
|
|
},
|
|
{
|
|
"chunk": 13,
|
|
"story": 1,
|
|
"summary": "John Hopkins seeks a romantic encounter, only to be kidnapped by a mysterious lady.",
|
|
"model_output": "John Hopkins seeks a romantic encounter, only to be kidnapped by a mysterious lady."
|
|
},
|
|
{
|
|
"chunk": 14,
|
|
"story": 1,
|
|
"summary": "Bess wants her St. Bernard, but John wants her to stop reading dangerous novels.",
|
|
"model_output": "Bess wants her St. Bernard, but John wants her to stop reading dangerous novels."
|
|
},
|
|
{
|
|
"chunk": 15,
|
|
"story": 2,
|
|
"summary": "Masie, a young saleslady, seeks to navigate the complexities of human nature while hiding her true identity.",
|
|
"model_output": "Masie, a young saleslady, seeks to navigate the complexities of human nature while hiding her true identity."
|
|
},
|
|
{
|
|
"chunk": 16,
|
|
"story": 2,
|
|
"summary": "The shopgirl Masie confronts the floorwalker Irving Carter with her flirtatious gaze, testing his resolve.",
|
|
"model_output": "The shopgirl Masie confronts the floorwalker Irving Carter with her flirtatious gaze, testing his resolve."
|
|
},
|
|
{
|
|
"chunk": 17,
|
|
"story": 2,
|
|
"summary": "Irving Carter desires to charm a shopgirl despite feeling out of place.",
|
|
"model_output": "Irving Carter desires to charm a shopgirl despite feeling out of place."
|
|
},
|
|
{
|
|
"chunk": 18,
|
|
"story": 2,
|
|
"summary": "Willie nervously asked for her company, hoping she'd agree to a date.",
|
|
"model_output": "Willie nervously asked for her company, hoping she'd agree to a date."
|
|
},
|
|
{
|
|
"chunk": 19,
|
|
"story": 2,
|
|
"summary": "Carter seeks love but fears rejection, while Masie uses him to prove her worth.",
|
|
"model_output": "Carter seeks love but fears rejection, while Masie uses him to prove her worth."
|
|
},
|
|
{
|
|
"chunk": 20,
|
|
"story": 2,
|
|
"summary": "Masie faces Carter's marriage proposal, wanting to see through his sincerity.",
|
|
"model_output": "Masie faces Carter's marriage proposal, wanting to see through his sincerity."
|
|
},
|
|
{
|
|
"chunk": 21,
|
|
"story": 2,
|
|
"summary": "Masie looked up at Carter, her eyes alight with a flicker of possibility.",
|
|
"model_output": "Masie looked up at Carter, her eyes alight with a flicker of possibility."
|
|
},
|
|
{
|
|
"chunk": 22,
|
|
"story": 2,
|
|
"summary": "She wants to end their secret affair, but he feels a hidden joy.",
|
|
"model_output": "She wants to end their secret affair, but he feels a hidden joy."
|
|
},
|
|
{
|
|
"chunk": 23,
|
|
"story": 3,
|
|
"summary": "Big Jim Dougherty sought acceptance in Manhattan's titled nobility of sportdom.",
|
|
"model_output": "Big Jim Dougherty sought acceptance in Manhattan's titled nobility of sportdom."
|
|
},
|
|
{
|
|
"chunk": 24,
|
|
"story": 3,
|
|
"summary": "He seeks a private life away from his public Oriental sport, yet feels trapped in public life.",
|
|
"model_output": "He seeks a private life away from his public Oriental sport, yet feels trapped in public life."
|
|
},
|
|
{
|
|
"chunk": 25,
|
|
"story": 3,
|
|
"summary": "Big Jim\" expected her to ask for a dinner date, not insist on one.",
|
|
"model_output": "\"Big Jim\" expected her to ask for a dinner date, not insist on one."
|
|
},
|
|
{
|
|
"chunk": 26,
|
|
"story": 3,
|
|
"summary": "Big Jim Dougherty wanted to hide his wife's presence, but a mysterious bird-like being at her side made him change plans.",
|
|
"model_output": "Big Jim Dougherty wanted to hide his wife's presence, but a mysterious bird-like being at her side made him change plans."
|
|
},
|
|
{
|
|
"chunk": 27,
|
|
"story": 3,
|
|
"summary": "Big Jim seeks to impress his wife at a fancy restaurant, hoping her beauty and charm will win him praise.",
|
|
"model_output": "Big Jim seeks to impress his wife at a fancy restaurant, hoping her beauty and charm will win him praise."
|
|
},
|
|
{
|
|
"chunk": 28,
|
|
"story": 3,
|
|
"summary": "The honorable Patrick Corrigan seeks to expose a secret and win Mrs. Dougherty's heart, while \"Big Jim\" struggles to keep up.",
|
|
"model_output": "The honorable Patrick Corrigan seeks to expose a secret and win Mrs. Dougherty's heart, while \"Big Jim\" struggles to keep up."
|
|
},
|
|
{
|
|
"chunk": 29,
|
|
"story": 3,
|
|
"summary": "Patrick seeks to impress Mrs. Dougherty with a bold gesture, but is thwarted by her quicker action.",
|
|
"model_output": "Patrick seeks to impress Mrs. Dougherty with a bold gesture, but is thwarted by her quicker action."
|
|
},
|
|
{
|
|
"chunk": 30,
|
|
"story": 3,
|
|
"summary": "The bride desires her husband's affection, yet fears losing him to distant admirers.",
|
|
"model_output": "The bride desires her husband's affection, yet fears losing him to distant admirers."
|
|
},
|
|
{
|
|
"chunk": 31,
|
|
"story": 3,
|
|
"summary": "The Kid seeks a peach but finds oranges instead, rushing to a fruiterer just as the lights go out.",
|
|
"model_output": "The Kid seeks a peach but finds oranges instead, rushing to a fruiterer just as the lights go out."
|
|
},
|
|
{
|
|
"chunk": 32,
|
|
"story": 3,
|
|
"summary": "Kid seeks to infiltrate Denver Dick's gambling den but finds it closed, needing help from the captain.",
|
|
"model_output": "Kid seeks to infiltrate Denver Dick's gambling den but finds it closed, needing help from the captain."
|
|
},
|
|
{
|
|
"chunk": 33,
|
|
"story": 3,
|
|
"summary": "The captain seeks to lead a raid, but doubts success as the Kid's plan unfolds.",
|
|
"model_output": "The captain seeks to lead a raid, but doubts success as the Kid's plan unfolds."
|
|
},
|
|
{
|
|
"chunk": 34,
|
|
"story": 3,
|
|
"summary": "The Kid seeks a last peach at a lavish dinner, fearing theft, while authorities arrive.",
|
|
"model_output": "The Kid seeks a last peach at a lavish dinner, fearing theft, while authorities arrive."
|
|
},
|
|
{
|
|
"chunk": 35,
|
|
"story": 3,
|
|
"summary": "The Kid seeks to please his bride by delivering a requested peach, despite its trivial nature.",
|
|
"model_output": "The Kid seeks to please his bride by delivering a requested peach, despite its trivial nature."
|
|
},
|
|
{
|
|
"chunk": 36,
|
|
"story": 4,
|
|
"summary": "Stuck in stone walls, city man craves nature's first signs to confirm spring's arrival.",
|
|
"model_output": "Stuck in stone walls, city man craves nature's first signs to confirm spring's arrival."
|
|
},
|
|
{
|
|
"chunk": 37,
|
|
"story": 4,
|
|
"summary": "Mr. Peters wants the dollar his wife has, but fears losing face among friends.",
|
|
"model_output": "Mr. Peters wants the dollar his wife has, but fears losing face among friends."
|
|
},
|
|
{
|
|
"chunk": 38,
|
|
"story": 4,
|
|
"summary": "Maguire wants the dollar, but Mr. Peters plans to get the case note instead.",
|
|
"model_output": "Maguire wants the dollar, but Mr. Peters plans to get the case note instead."
|
|
},
|
|
{
|
|
"chunk": 39,
|
|
"story": 4,
|
|
"summary": "Mr. Peters schemes to charm Mrs. Peters away from a tea job, while she resists with a tough stance.",
|
|
"model_output": "Mr. Peters schemes to charm Mrs. Peters away from a tea job, while she resists with a tough stance."
|
|
}
|
|
]
|
|
} |