Jesse.Markowitz/TheLadder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace clustering with a catalogue, and hand-write the references to judge it against

Browse Source 
The pipeline's embed-and-cluster step is dead, and this commit holds both the
evidence for that and the step proposed to replace it.

Predicaments. Scenes are re-described as "what the person is up against", with
no names, jobs or places, then embedded and clustered (redescribe.py,
topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading
both side by side. Two defects the pilot exposed are fixed: split.py missed
titles in quotes and a contents subtitle after a dash, so three stories had been
merged into their neighbours, and strip_names.py read New York place names as
people. The corrected corpus is probe/v2 (97 stories, 839 scenes);
carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from
20% to 13% at k=60, short of the pre-registered 10%.

Hand references. Three corpora were read scene by scene and written up by hand,
under the same prompt rules the local models get, as a baseline to judge them
against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's
Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of
the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a
readable page. No inference was used for any of them.

Catalogue. probe/catalogue maps every hand group in the three references onto 36
situation entries, with an answer key per corpus and one recurrence rule applied
to all three. classify.py assigns a scene one entry or none, leave-one-corpus-
out; score.py checks it against the key, with a self-test on random labels.

Why clustering is out: hand-written predicaments, embedded and clustered exactly
as the model's were, agree with the hand grouping at ARI 0.05 — no better than
the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even
shortlist: the hand label is the nearest entry 13% of the time and in the top 8
half the time.

The classification runs are not here. The dev and test runs are pre-registered
in probe/catalogue/README.md with the bar set beforehand, and are blocked on the
inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene
partial output in out/ is not a result.

Review page. The situation review is now a browser page rather than JSON edited
by hand (review_page.py, review_page_logic.cjs with Node tests, format schema
v2). It has never been rendered in a real browser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
This commit is contained in:
JesseMarkowitz
2026-09-20 17:19:42 -04:00
co-authored by Claude Opus 5
parent fa3769d0fe
commit e5617b86ba
135 changed files with 82027 additions and 1778 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/story-to-pack/probe/catalogue/README.md
+108
View File
@@ -0,0 +1,108 @@
# Situation catalogue: classify, don't cluster
Built 2026-09-17 from the three hand references: O. Henry (`../v2/claude/`), Wharton (`../wharton/claude/`)
and Jacobs (`../jacobs/claude/`). This is the step that replaces embed-and-cluster in the proposed pipeline:
a model assigns each scene's predicament one catalogue entry, or `none`.
## Files
| File | What |
| --- | --- |
| `catalogue_src.py` | The 36 entries (definition, includes, excludes, tone, recurs) and `SOURCES`, which maps every hand group in the three references to exactly one entry. Running it checks the mapping and writes the rest. No inference. |
| `catalogue.json` | The entries, with up to three worked examples per corpus, each tagged with its corpus |
| `labels-<corpus>.json` | Every scene's catalogue label from the hand grouping, or `none`. This is the answer key for scoring |
| `recurrence.log` | One recurrence rule applied to all three: at least 5 scenes, at least max(3, 4% of the stories) stories, and no story over 50% |
| `classify.py` | Classifies one scene per request through `../ollama.py`. Leave-one-corpus-out: the examples never come from the corpus being classified. `--k 0` offers the whole catalogue |
| `score.py` | Scores classifier output against `labels-*.json`. `--selftest` checks the scorer on hand-vs-hand and on random labels |
| `emb/` | Cached nomic-embed-text vectors for the catalogue and the O. Henry 14B predicaments |
| `out/` | Classifier output |
## Results so far
**1. One rule for all three corpora** (`recurrence.log`). O. Henry keeps 16 recurring situations and
Wharton keeps 16. Jacobs falls from its hand-counted 19 to **6**, because its groups had used a two-story
minimum. With 12 stories, three stories a group is the least that means "recurs", and most of its
groups do not reach it. Only `rival`, `strings` and `newcomer` are found in all three corpora; ten more
are in two. A 20-30 story corpus is a real floor.
**2. Clustering fails even on the best text.** My 839 hand-written O. Henry predicaments were
embedded exactly as `embed.py` embedded the 14B ones (`../v2/claude/predicament_embeddings-claude.json`),
and the same k = 60 k-means was run on them (`../v2/claude/compare-claude-embeddings.log`). The groups
agree with my own hand grouping at **ARI 0.05** (0.05-0.05 over four seeds), no better than the 14B text's
0.07. The fault is the embed-and-cluster step, not the predicament text. Better rewriting will not
rescue clustering.
**3. Embeddings cannot even shortlist.** With the 14B predicaments against the catalogue definitions, the
hand label is the nearest entry for 13% of situated scenes, is in the top 8 for 50%, and is in the top 20
for 83%. So `classify.py` now offers the whole catalogue (`--k 0`), with the catalogue in the system prompt
so the server can reuse the cached prefix.
**4. Classifier pilot: 40 scenes only, too small to judge** (`pilot40.log`; qwen3:14b, shortlist k = 8,
14B predicaments). The model called 34 of 40 scenes situated where the hand labels have 14. Where both
called a scene situated, its label was right in 3 of 13 (23%). Kappa was 0.09. The prompt has since been
changed: the whole catalogue, a note to expect about half `none`, and a fit rating from 1 to 3 that
`score.py --minfit=` can threshold. The changed prompt has not been measured.
**4b. The changed prompt, on 30 scenes before the host hung** (`out/ohenry-q14-dev1.partial.json`, scored
in `pilot40.log`): it tells situated scenes from `none` better (precision 67%, recall 100%, kappa 0.23), but
where both call a scene situated, its label is right in only 2 of 14. Reading the confusions, some of its
labels are defensible. I labelled a whole story arc with one situation (every scene of *A Lickpenny Lover* is
`station`), while the model judges each scene alone (`pretence`, `unspoken`). And the 14B predicaments
("A man struggles to reconcile his pride with...") are too vague to classify from. So the next development
round should (a) also give the classifier the scene summary, or the predicaments either side, and (b) score
at the level a pack uses: which situations recur, with what members, rather than scene-exact agreement.
## Not done: the inference host's GPU fell off the bus
At **22:19:07 on 2026-09-17**, the inference host's GPU fell off the PCIe bus (kernel: `Xid 79`, then `Xid 154, GPU Reset
Required`). It happened in the middle of a `classify.py` development request whose prompt was about 2,450 tokens (the
whole catalogue). Over the ten minutes before, calls had slowed from about 2 s a scene to about 20 s. This is the
second such fault; the first was on 2026-09-13. **No power or link loggers were running**, because SSH was refused
while the key was out of the agent, so there is no evidence either way on power. After the fault, Ollama carried on
on the CPU. That is why nomic-embed-text still answered, and finding 2 was computed on CPU embeddings, which does
not change the method. All client processes were stopped. The host needs a reboot (reset required). The NVIDIA
driver asks for `nvidia-bug-report.sh` to be run as root before the module is unloaded.
The prompt design matters for this host: offering the whole catalogue makes every call a long prefill. Either keep
the catalogue as a cached prefix (the current `classify.py`, which puts it in the system prompt; untested on the
GPU), or shortlist with something better than embeddings, or run with a lower power limit and the loggers on.
### Third fault, with the loggers running: 2026-09-18 01:00:03
After a reboot, with the power limit set to 200 W (from 280 W) and all three loggers running, a 20-scene pilot
(`classify.py ohenry q14 --limit 20 --tag=-dev1`, catalogue as the cached system prompt) lost the GPU about 15 s in.
The logs are in `gpu-logs/*-2026-09-18-0056.*`. From the link/power CSV:
| Time | Power | Utilisation | Notes |
| --- | --- | --- | --- |
| 00:59:54 | 309.31 W | 49% | enforced limit 200 W; throttle reason 0x4 (SW power cap) |
| 00:59:55 to 00:59:59 | about 178 to 202 W | 100% | sustained prefill |
| 01:00:02 | 174.96 W | | last good sample |
| 01:00:03 | | | "GPU is lost"; dmon shows `-` from here |
The temperature never went above 39 °C, so the cause is not thermal. The link was PCIe gen 3 x4. The card drew
309 W for a sample despite a 200 W cap, and then dropped after about ten seconds of load at the cap. That points to
power delivery to the card or the OCuLink dock, not to the prompt or the software. A lower cap did not prevent it.
**No more classification runs on that host until its hardware has been looked at.** The pilot output is not a result.
`gpu-logs/nvidia-bug-report-2026-09-18.log.gz` is the driver's report, taken after the fault. The kernel log in it
shows no AER or PCIe error messages at any fault. That is not evidence of a clean link: at boot the kernel logs `_OSC: platform does not support [SHPCHotplug AER LTR DPC]`, so the firmware never gives the OS AER control and link errors would not be reported. There
was an `Xid 32` (corrupted push-buffer stream) from Ollama at 00:54:48, five minutes before this drop, and an `Xid 13`
graphics exception on 2026-09-13, before the first drop.
## Next, once the host is healthy
The plan below is fixed before any of the runs, so they cannot be tuned after the fact.
- **Development slice:** O. Henry scenes 0-119. Tune the prompt and the `--minfit` threshold only here.
(`classify.py ohenry q14 --limit 120 --tag=-dev1`)
- **Test:** O. Henry scenes 120-838, run once (`--start 120 --tag=-test`). Score with
`score.py out/ohenry-q14-test.json --range=120:839`. Then run the same prompt on the hand-written
predicaments (`ohenry claude`) to separate rewrite quality from classification quality. Then run
`wharton claude` and `jacobs claude`, which have no local-model predicaments yet, to see how the
catalogue transfers.
- **What counts as success** is precision per situation, not the number of situations that pass:
`--selftest` shows random labels also "pass" all 16. The bar is set before the test run: for the
situations both hand and model call recurring, a median model precision of at least 60%.
- At about 2 s a scene, O. Henry takes about 30 minutes a text set. That is a long run, so start the
loggers first.
tools/story-to-pack/probe/catalogue/catalogue.json
+1270
View File
File diff suppressed because it is too large Load Diff
tools/story-to-pack/probe/catalogue/catalogue_src.py
+271
View File
@@ -0,0 +1,271 @@
"""The situation catalogue: a closed list a model classifies scenes against, instead of clustering them.
Seeded from three hand-made references (O. Henry ../v2/claude, Wharton ../wharton/claude, Jacobs ../jacobs/claude).
Every hand group in those references maps to exactly one catalogue entry (SOURCES). Running this file:
1. checks the mapping is total and every entry has at least one source,
2. writes catalogue.json (entries + worked examples, tagged by corpus so a classifier can leave a corpus out),
3. writes labels-<corpus>.json: every scene's catalogue label from the hand grouping, or "none",
4. applies ONE recurrence rule to all three corpora and writes recurrence.log.
No inference.
"""
import collections, json, math, pathlib, sys
HERE = pathlib.Path(__file__).resolve().parent
PROBE = HERE.parent
E = lambda id, name, definition, includes, excludes, tone='any', recurs=True: dict(
id=id, name=name, definition=definition, includes=includes, excludes=excludes, tone=tone, recurs=recurs)
CATALOGUE = [
E('rival', 'A rival for the same person',
'Someone else wants the person you want, and only one of you can have them.',
'suitors competing; fear of losing a partner to someone else; temptation to strike at the rival',
'competing for money or a prize that is not a person (your_scheme)'),
E('strings', 'Help with strings attached',
'Help, money or a chance is offered to you, but only on terms that will cost you something you value.',
'family money conditional on obeying them; a backer who now wants something; a windfall with a hidden cost',
'the terms require doing something you think wrong (compromise_for_money); a stranger\'s trap (sure_thing)'),
E('compromise_for_money', 'Money for going along',
'You can have money, success or a meal if you go along with something you think wrong or false.',
'selling out your standards for pay; the client wants the flattering or fake version; a compromising offer when you are hungry',
'the cost is only inconvenient, not wrong (strings)'),
E('sure_thing', 'The sure thing',
'A plausible person offers an easy fortune that is probably a trap set for you.',
'confidence tricks aimed at you; too-good-to-be-true bargains; strangers who befriend you for your money',
'you are the one running the scheme (your_scheme)'),
E('your_scheme', 'Your scheme turns on you',
'You are running a scheme or quietly tipping a deal your way, and it may be about to turn on you.',
'cheating a partner; rigging a contest; a trick coming apart in front of the people you meant to fool',
'someone else is scheming against you (sure_thing)'),
E('after_your_money', 'People after your money',
'You have money, or are thought to, and people around you are suddenly courting it.',
'heirs circling; flatterers and hangers-on; pressure to reveal a hidden windfall',
'a single stranger\'s con (sure_thing)'),
E('past_comes_back', 'The past comes back',
'Someone from your past, or your partner\'s, turns up with a claim or a hold that threatens your present life.',
'blackmail over an old secret; a former spouse or intimate who still has a claim; an old associate who could expose you',
'you simply discover something a person close to you hid (what_they_hid)'),
E('hiding_from_close', 'Hiding it from someone close',
'You are hiding something, and someone close to you keeps innocently coming near it.',
'keeping a guilty or embarrassing secret from a spouse or family; near misses',
'the secret belongs to someone else (their_secret)'),
E('their_secret', 'Keeping someone else\'s secret',
'You have been drawn into another person\'s secret and must decide whether to lie, stay silent or give it up.',
'covering for a friend or employer; being questioned by a suspicious superior',
'your own secret (hiding_from_close)'),
E('what_they_hid', 'What they hid',
'You discover that a person close to you has been hiding something, and must decide what it changes.',
'a partner\'s concealed past or disappointment; a friend\'s lie revealed',
'the hidden thing arrives as a claim or threat from outside (past_comes_back)'),
E('pretence', 'Passing for more than you are',
'You are pretending to be more, or other, than you are, to impress someone, and may be found out.',
'dressing and talking above your station; invented titles or histories; a thrilling story nobody can check',
'you are genuinely above or below the other person and must decide what to show (station)'),
E('caught_out', 'Caught out',
'You have been caught, or are about to be seen, doing or being what you meant nobody to see.',
'a private arrangement about to be judged publicly; caught somewhere you should not be; a joke taken in earnest',
'you are blamed for something you did not do (wrongly_judged)'),
E('wrongly_judged', 'What it looked like',
'Someone whose opinion matters has judged you by appearances, for something you did not do, and will not hear you out.',
'an innocent moment misread; suspected of a theft; taken for a fraud because of how you look',
'you really did it (caught_out)'),
E('let_another_take_blame', 'Someone else is taking the blame',
'Suspicion has fallen on someone innocent for what you did, and you could let it stay there.',
'an innocent person punished for your act; a stranger\'s blunder gives you a scapegoat',
'you are the innocent one (wrongly_judged)'),
E('cant_prove_it', 'Sure, but cannot prove it',
'You are sure who wronged you, and you cannot prove it.',
'accusing without evidence; authorities brushing you off; trying to trap the suspect',
'you are the one accused (wrongly_judged)'),
E('station', 'Across the gap in station',
'Someone well above or below you in station wants you, and each of you must decide how much to show of where you really stand.',
'courting or marrying across class; ashamed of your origins before a grander partner; a rich suitor not believed',
'pretending to a station you do not have (pretence)'),
E('home_unseen', 'Unseen at home',
'You and a partner at home have stopped really seeing each other, and one of you wants to be noticed.',
'neglect; taking a spouse for granted; a partner whose blind approval or coarseness makes nothing matter',
'open quarrels about blame (peace_at_home)'),
E('peace_at_home', 'Keeping the peace at home',
'Someone at home blames you or demands their way, and you must decide whether to give way to keep the peace.',
'being held responsible for household trouble; backing down before a spouse or elder; household quarrels',
'loneliness without conflict (home_unseen); an outsider in the house (unwelcome_in_home)'),
E('unwelcome_in_home', 'An unwelcome guest',
'Someone you do not want has settled into your home, and you cannot seem to make them go.',
'a relative or intruder who stays; having to make polite room for someone who holds power over the household',
'a former partner\'s claim (past_comes_back)'),
E('power_over_you', 'Under someone\'s thumb',
'Someone with power over you treats you with contempt, or must be humoured though you resent them, and you want respect or to turn the tables.',
'a superior who humiliates you; having to be civil and helpful to someone you fear or dislike because of your position',
'a well-meaning benefactor (benefactor)'),
E('benefactor', 'Help nobody asked for',
'Someone with money wants to do good to someone below them on their own terms: either you are offered such help, or you are the one offering it.',
'charity with a sermon or a price in pride; reformers who stop you working; a rich person seeking gratitude or restitution',
'a conditional offer between equals (strings)'),
E('patron', 'A handshake from the top',
'Someone powerful could be seen with you, and getting that notice will cost you.',
'currying favour with a great man; spending to be seen with the important; wanting a patron at your wedding',
'being humiliated by them (power_over_you)'),
E('pressed_to_wrong', 'Pressed to do wrong',
'Someone is pressing you to do something you believe is wrong or dangerous, including turning on a friend you owe.',
'pressure to testify against or turn in a friend; a loved one begging you to do something wicked; being invited into a crooked scheme',
'you are merely paid to compromise your standards (compromise_for_money)'),
E('go_between', 'Carrying someone else\'s case',
'You are asked to carry another person\'s claim or warning to a third party, knowing it may cost you with one side or the other.',
'pleading a case you doubt; passing on an unwelcome message; warning an outsider about a relative',
'keeping a secret (their_secret)'),
E('sacrifice', 'Giving it up for them',
'You can help or protect someone you care for only by giving up something of your own, and they may never know.',
'selling a treasure for a gift; secret humble work; giving your savings or name away for another',
'being paid to give something up (strings)'),
E('owning_up', 'Owning up',
'You once wronged someone, and now face them, or a stand-in, and must decide how much to admit or how to make amends.',
'confession; seeking forgiveness; a public show of devotion to win someone back',
'an innocent is blamed right now for what you did (let_another_take_blame)'),
E('own_words_back', 'Held to your own words',
'A principle, promise or rule you once used for your own ends is now turned against you.',
'hoist by your own code; a promise you never expected to be called in', '', recurs=False),
E('newcomer', 'New to the place',
'You are new to a place or household that ignores you, warns you, or leaves you lost, and you must find your footing.',
'arriving in the city; a new post with warnings about the person in charge; waking somewhere strange',
'news from the place you left (homecoming)'),
E('homecoming', 'Word from home',
'The people you left want you back, and word of them has reached you.',
'a messenger from home; a parent waiting; a reunion with an old friend from where you came from',
'a claim or threat from the past (past_comes_back)'),
E('dread', 'Something in the dark',
'Something frightening may be close by, and you must decide whether to face it.',
'fear of what a wish or deed has brought; noises in the dark; a thing you must undo before it arrives',
'ordinary social danger', tone='horror'),
E('waiting', 'Waiting on their decision',
'Your hopes hang on a decision entirely in another\'s hands, and nothing you do seems to hurry it.',
'waiting for a proposal, a verdict or a signal; pretending indifference while it hangs',
'you are the one who cannot speak (unspoken)'),
E('unspoken', 'You cannot say it',
'Someone you want is waiting for you to say it, and you cannot find the words or the moment.',
'shy lovers; a single short chance to speak; feelings hidden behind a business arrangement',
'waiting for the other to decide (waiting)'),
E('overlooked', 'Beside the admired one',
'You are the plain or overlooked one beside someone admired, and want, once, to be noticed.',
'the plain co-worker; going to dances on someone else\'s arm',
'a partner at home who ignores you (home_unseen)'),
E('sign', 'A sign',
'Someone in a bad way wants a sign to tell them what to do, and one seems to have turned up.',
'fortune-tellers, omens, mysterious messages, reading a coincidence as meant for you',
'real supernatural danger (dread)'),
E('despised', 'The one nobody thinks much of',
'Someone everybody looks down on wants a chance, and may be the only one who can help.',
'a tramp, a foreigner, a crook or a failure who turns out to be the helper',
'someone powerful (patron)'),
E('habit', 'The old habit',
'An old habit gives you away at exactly the moment you need to be someone new.',
'a former trade\'s manner resurfacing; an old craving at the worst moment',
'', recurs=False),
]
# every hand group in the three references -> one catalogue id
SOURCES = {
'ohenry': {'pose': 'pretence', 'benefactor': 'benefactor', 'taken_for_granted': 'home_unseen', 'station': 'station',
'newcomer': 'newcomer', 'easy_money': 'sure_thing', 'sacrifice': 'sacrifice', 'rival': 'rival',
'price': 'compromise_for_money', 'despised': 'despised', 'loyalty': 'pressed_to_wrong',
'appearances': 'wrongly_judged', 'sign': 'sign', 'strings': 'strings', 'honest_work': 'compromise_for_money',
'unspoken': 'unspoken', 'overlooked': 'overlooked', 'homecoming': 'homecoming', 'patron': 'patron', 'habit': 'habit'},
'wharton': {'tainted_money': 'compromise_for_money', 'strings_attached': 'strings', 'old_attachment': 'past_comes_back',
'what_they_hid': 'what_they_hid', 'keep_their_secret': 'their_secret', 'exposed': 'caught_out',
'civil_to_them': 'power_over_you', 'plead_their_case': 'go_between', 'own_words_back': 'own_words_back',
'give_it_up': 'sacrifice', 'keep_the_peace': 'peace_at_home', 'unseen_at_home': 'home_unseen',
'new_and_warned': 'newcomer', 'the_rival': 'rival', 'owning_up': 'owning_up', 'waiting_on_them': 'waiting'},
'jacobs': {'G01': 'past_comes_back', 'G02': 'hiding_from_close', 'G03': 'let_another_take_blame', 'G04': 'rival',
'G05': 'your_scheme', 'G06': 'your_scheme', 'G07': 'after_your_money', 'G08': 'strings', 'G09': 'your_scheme',
'G10': 'power_over_you', 'G11': 'unwelcome_in_home', 'G12': 'dread', 'G13': 'pressed_to_wrong',
'G14': 'pretence', 'G15': 'caught_out', 'G16': 'peace_at_home', 'G17': 'cant_prove_it', 'G18': 'newcomer',
'G19': 'owning_up'},
}
def load_corpus(name):
"""-> chunks, {chunk: predicament}, {chunk: hand group id}"""
if name == 'ohenry':
d = PROBE / 'v2'
sys.path.insert(0, str(d / 'claude')); from assign import ASSIGN
pred = {x['chunk']: x['summary'] for x in json.load(open(d / 'claude/predicaments-claude.json'))['summaries']}
return json.load(open(d / 'chunks.json')), pred, dict(ASSIGN)
d = PROBE / name
groups = json.load(open(d / 'claude/groups.json'))['groups']
hand = {}
for g in groups:
for m in g['members']:
c = m if isinstance(m, int) else m['chunk']
assert c not in hand, (name, c); hand[c] = g['id']
if name == 'wharton':
pred = {x['chunk']: x['summary'] for x in json.load(open(d / 'claude/predicaments.json'))['summaries']}
else:
rec = json.load(open(d / 'claude/records.json')); rec = rec if isinstance(rec, list) else rec['records']
pred = {r['chunk']: r['predicament'] for r in rec}
return json.load(open(d / 'chunks.json')), pred, hand
CORPORA = ['ohenry', 'wharton', 'jacobs']
# ---- one recurrence rule for all three --------------------------------------------------------------
def min_stories(n_stories): return max(3, math.ceil(0.04 * n_stories)) # 3 for 10-12 stories, 4 for 97
MIN_SCENES, MAX_DOMINANT = 5, 0.50
def recurs(members, story):
st = collections.Counter(story[c] for c in members)
return len(members) >= MIN_SCENES and len(st) >= min_stories(len(set(story))) and max(st.values()) / len(members) <= MAX_DOMINANT, len(st), (max(st.values()) / len(members) if members else 0)
if __name__ == '__main__':
ids = [e['id'] for e in CATALOGUE]
assert len(ids) == len(set(ids)), 'duplicate id'
problems = []
for corpus, m in SOURCES.items():
for g, c in m.items():
if c not in ids: problems.append(f'{corpus}:{g} -> unknown {c}')
used = {c for m in SOURCES.values() for c in m.values()}
problems += [f'catalogue entry {i} has no source group' for i in ids if i not in used]
for e in CATALOGUE:
for x in e['excludes'].split(';'):
for i in ids:
pass
examples = collections.defaultdict(list)
log = []
for corpus in CORPORA:
chunks, pred, hand = load_corpus(corpus)
unmapped = sorted({g for g in hand.values() if g not in SOURCES[corpus]})
if unmapped: problems.append(f'{corpus}: hand groups not in SOURCES: {unmapped}')
labels = {i: SOURCES[corpus].get(hand[i], 'none') if i in hand else 'none' for i in range(len(chunks))}
(HERE / f'labels-{corpus}.json').write_text(json.dumps(
{'corpus': corpus, 'from': 'hand groups mapped through catalogue_src.SOURCES',
'labels': [{'chunk': i, 'story': chunks[i]['story'], 'label': labels[i], 'predicament': pred[i]} for i in range(len(chunks))]},
indent=0, ensure_ascii=False))
# examples: up to 3 per (entry, corpus), spread across stories, taken from the hand members
per = collections.defaultdict(list)
for i in sorted(labels):
if labels[i] != 'none': per[labels[i]].append(i)
for lab, mem in per.items():
seen = set()
for i in mem:
if chunks[i]['story'] in seen: continue
seen.add(chunks[i]['story']); examples[lab].append({'corpus': corpus, 'chunk': i, 'predicament': pred[i]})
if len(seen) == 3: break
story = [c['story'] for c in chunks]
ns = len(set(story))
log.append(f'\n== {corpus}: {ns} stories, {len(chunks)} scenes; rule: >= {MIN_SCENES} scenes, >= {min_stories(ns)} stories, no story over {MAX_DOMINANT:.0%}')
log.append(f' scenes labelled: {sum(l != "none" for l in labels.values())} ({sum(l != "none" for l in labels.values())/len(chunks):.0%})')
passing = []
for lab in sorted(per, key=lambda k: -len(per[k])):
ok, nst, dom = recurs(per[lab], story)
passing += [lab] if ok else []
log.append(f' {"PASS" if ok else " "} {lab:24s} {len(per[lab]):3d} scenes {nst:3d} stories top story {dom:4.0%}')
log.append(f' -> {len(passing)} recurring situations of {len(per)} present')
cover = collections.defaultdict(set)
for corpus, m in SOURCES.items():
for c in m.values(): cover[c].add(corpus)
log.append('\n== catalogue coverage: entries found in 3 / 2 / 1 corpora')
for k in (3, 2, 1):
log.append(f' {k}: ' + ', '.join(i for i in ids if len(cover[i]) == k))
if problems:
print('\n'.join(problems)); sys.exit(1)
out = {'about': 'Situation catalogue for story-to-pack, seeded by hand from O. Henry, Wharton and Jacobs references (2026-09-17). '
'A classifier assigns each scene predicament one entry or "none". Examples carry their corpus so a test '
'on one corpus can use only the others.',
'entries': [dict(e, corpora=sorted(cover[e['id']]), examples=examples[e['id']]) for e in CATALOGUE]}
(HERE / 'catalogue.json').write_text(json.dumps(out, indent=1, ensure_ascii=False))
(HERE / 'recurrence.log').write_text('\n'.join(log).lstrip() + '\n')
print(f'catalogue.json: {len(CATALOGUE)} entries'); print('\n'.join(log))
tools/story-to-pack/probe/catalogue/classify.py
+107
View File
@@ -0,0 +1,107 @@
"""Classify scene predicaments against the catalogue, one scene at a time.
STP_OLLAMA=http://<host>:11434 python3 classify.py <corpus> <text-set> [--limit N]
corpus ohenry | wharton | jacobs
text-set claude (the hand-written predicaments) | q14 (qwen3:14b predicaments, O. Henry only)
Leave-one-corpus-out: the worked examples shown never come from the corpus being classified, so this
tests the catalogue on a corpus it has not seen. Embeddings (nomic-embed-text) only narrow the choice:
the scene's K nearest entries, plus "none", go to the model. Checkpoints after every scene; resumes.
Writes out/<corpus>-<text-set>.json.
"""
import argparse, json, math, pathlib, re, sys, time
HERE = pathlib.Path(__file__).resolve().parent
PROBE = HERE.parent
sys.path.insert(0, str(PROBE))
import ollama
ap = argparse.ArgumentParser()
ap.add_argument('corpus'); ap.add_argument('textset')
ap.add_argument('--limit', type=int); ap.add_argument('--k', type=int, default=0, help='0 = offer the whole catalogue')
ap.add_argument('--start', type=int, default=0)
ap.add_argument('--tag', default='')
ap.add_argument('--model', default='qwen3:14b')
args = ap.parse_args()
EMB = 'nomic-embed-text'
cat = json.load(open(HERE / 'catalogue.json'))['entries']
labels = json.load(open(HERE / f'labels-{args.corpus}.json'))['labels']
if args.textset == 'claude':
texts = [x['predicament'] for x in labels]
elif args.textset == 'q14':
assert args.corpus == 'ohenry'
q = {x['chunk']: x['summary'] for x in json.load(open(PROBE / 'v2/predicaments_clean.json'))['summaries']}
texts = [q[x['chunk']] for x in labels]
else:
sys.exit('text-set must be claude or q14')
def norm(v):
m = math.sqrt(sum(x * x for x in v)) or 1.0
return [x / m for x in v]
# embeddings, cached
def cached_embed(name, items):
p = HERE / 'emb' / f'{name}.json'; p.parent.mkdir(exist_ok=True)
if p.exists():
d = json.load(open(p))
if d['texts'] == items: return d['vectors']
vecs = []
for i in range(0, len(items), 16):
vecs += ollama.embed(EMB, ['search_query: ' + t if name != 'catalogue' else 'search_document: ' + t for t in items[i:i + 16]])
p.write_text(json.dumps({'model': EMB, 'texts': items, 'vectors': vecs}))
return vecs
entry_text = [f"{e['name']}. {e['definition']} Includes: {e['includes']}." for e in cat]
E = [norm(v) for v in cached_embed('catalogue', entry_text)]
S = [norm(v) for v in cached_embed(f'{args.corpus}-{args.textset}', texts)]
def examples_for(e):
ex = [x for x in e['examples'] if x['corpus'] != args.corpus][:2]
return ''.join(f'\n e.g. "{x["predicament"]}"' for x in ex)
SYSTEM = """You sort scenes from short stories into recurring SITUATIONS for a life-simulation game.
You get one scene, described as its main person's predicament, and a short list of candidate situations.
Answer with the ONE candidate that the predicament is an instance of, or "none".
Choose "none" when the scene is narration, scene-setting, a description of a place or type of person, or a
one-off turn that is not really any of the candidates. A shared word or setting is not enough: the
predicament itself -- what the person is up against and what they want -- must match the situation.
Most scenes in a story are NOT one of these situations: expect about half to be "none".
Also rate how clearly the predicament is an instance of your choice: 3 = plainly this situation,
2 = arguably, 1 = only loosely.
Reply with JSON only: {"label": "<candidate id or none>", "fit": <1, 2 or 3>}"""
out_path = HERE / 'out' / f'{args.corpus}-{args.textset}{args.tag}.json'; out_path.parent.mkdir(exist_ok=True)
state = json.load(open(out_path)) if out_path.exists() else {'model': args.model, 'k': args.k, 'results': []}
done = {r['chunk'] for r in state['results']}
n = len(texts) if not args.limit else min(args.limit, len(texts))
t0 = time.time(); new = 0
for i in range(args.start, n):
if labels[i]['chunk'] in done: continue
order = sorted(range(len(cat)), key=lambda j: -sum(a * b for a, b in zip(S[i], E[j])))
sims = order[:args.k] if args.k else list(range(len(cat)))
cands = '\n'.join(f"- {cat[j]['id']}: {cat[j]['definition']}{examples_for(cat[j])}" for j in sims)
# candidates go in the system prompt, identical from scene to scene when the whole catalogue is offered,
# so the server can reuse its cached prefix; the predicament comes last
system = SYSTEM + '\n\nCandidates:\n' + cands + '\n- none: not one of these situations'
user = f'Predicament: "{texts[i]}"'
raw, _, _ = ollama.chat(args.model, system, user, num_ctx=4096, num_predict=40, fmt='json',
think=False if args.model.startswith('qwen3') else None)
try:
j = json.loads(raw); lab = str(j.get('label', '')).strip(); fit = j.get('fit')
except Exception:
m = re.search(r'"label"\s*:\s*"([^"]+)"', raw); lab = m.group(1) if m else ''; fit = None
valid = {cat[j]['id'] for j in sims} | {'none'}
state['results'].append({'chunk': labels[i]['chunk'], 'label': lab if lab in valid else 'none',
'fit': fit, 'raw': raw if lab not in valid else None, 'candidates': [cat[j]['id'] for j in sims],
'nearest': cat[sims[0]]['id']})
new += 1
if new % 10 == 0 or i == n - 1:
out_path.write_text(json.dumps(state, indent=0))
el = time.time() - t0
print(f' {len(state["results"])}/{n} {el:.0f}s {el / new:.2f}s/scene', flush=True)
out_path.write_text(json.dumps(state, indent=0))
print('done', out_path.name, len(state['results']))
tools/story-to-pack/probe/catalogue/emb/catalogue.json
+1
View File
File diff suppressed because one or more lines are too long
tools/story-to-pack/probe/catalogue/emb/ohenry-q14.json
+1
View File
File diff suppressed because one or more lines are too long
tools/story-to-pack/probe/catalogue/labels-jacobs.json
+948
View File
@@ -0,0 +1,948 @@
{
"corpus": "jacobs",
"from": "hand groups mapped through catalogue_src.SOURCES",
"labels": [
{
"chunk": 0,
"story": 0,
"label": "peace_at_home",
"predicament": "Someone waiting up for a relative who comes in late wants to reproach him, and gets cheek back."
},
{
"chunk": 1,
"story": 0,
"label": "peace_at_home",
"predicament": "A man learns a relative has promised a stranger his hospitality without asking, and fears what his spouse will say."
},
{
"chunk": 2,
"story": 0,
"label": "unwelcome_in_home",
"predicament": "A man who wants to turn an unwelcome guest away cannot find the nerve to say it."
},
{
"chunk": 3,
"story": 0,
"label": "peace_at_home",
"predicament": "A man who has let something happen under his roof fears his spouse will hold him responsible for it."
},
{
"chunk": 4,
"story": 0,
"label": "rival",
"predicament": "A young man who brought the person he is courting along is pushed aside by an older man who takes charge of her."
},
{
"chunk": 5,
"story": 0,
"label": "rival",
"predicament": "A jealous suitor tries to put his rival in his place, and only makes himself look childish."
},
{
"chunk": 6,
"story": 0,
"label": "none",
"predicament": "A spurned man realises he holds something over the rival who humiliated him, and decides to use it."
},
{
"chunk": 7,
"story": 0,
"label": "hiding_from_close",
"predicament": "A man hiding an innocent but compromising secret is met by his spouse, while the one person who could clear him refuses to."
},
{
"chunk": 8,
"story": 0,
"label": "hiding_from_close",
"predicament": "A man tries to hurry his spouse away from a secret while someone signals behind his back that something is up."
},
{
"chunk": 9,
"story": 0,
"label": "caught_out",
"predicament": "A man caught with an innocent secret is disowned by the one person who could explain it."
},
{
"chunk": 10,
"story": 0,
"label": "let_another_take_blame",
"predicament": "A man whose spite has got an innocent person abused must decide whether to own up and defend her."
},
{
"chunk": 11,
"story": 0,
"label": "owning_up",
"predicament": "A man tries to win back someone he wronged with a dangerous public show of devotion."
},
{
"chunk": 12,
"story": 0,
"label": "your_scheme",
"predicament": "A man pretends to be in danger to win sympathy, and the one who saved him sees through it."
},
{
"chunk": 13,
"story": 0,
"label": "owning_up",
"predicament": "Two men who quarrelled over a woman are both left out in the cold by the women they wronged."
},
{
"chunk": 14,
"story": 1,
"label": "caught_out",
"predicament": "A man losing a friendly contest tries to distract from his mistake, and is caught at it by his family."
},
{
"chunk": 15,
"story": 1,
"label": "none",
"predicament": "A guest who wants to leave a dangerous subject alone is coaxed by his hosts' curiosity into bringing it out."
},
{
"chunk": 16,
"story": 1,
"label": "strings",
"predicament": "Someone offered a thing that grants wishes is warned by the man who used it that it brings sorrow."
},
{
"chunk": 17,
"story": 1,
"label": "strings",
"predicament": "A man is offered a dangerous gift by someone who begs him to destroy it, and wants to keep it anyway."
},
{
"chunk": 18,
"story": 1,
"label": "strings",
"predicament": "A man half-believing in a promise of easy money must decide whether to test it and what to ask for."
},
{
"chunk": 19,
"story": 1,
"label": "dread",
"predicament": "A family that tempted fate laughs off their fears in daylight while one of them cannot shake them."
},
{
"chunk": 20,
"story": 1,
"label": "strings",
"predicament": "A family waiting on a hoped-for windfall sees a stranger approach who might bring it, or bad news."
},
{
"chunk": 21,
"story": 1,
"label": "none",
"predicament": "Parents receive news that their only child has been killed, from someone who cannot bring himself to say it."
},
{
"chunk": 22,
"story": 1,
"label": "strings",
"predicament": "A man learns the money he wished for has come as compensation for his child's death."
},
{
"chunk": 23,
"story": 1,
"label": "pressed_to_wrong",
"predicament": "A grieving woman seizes on a desperate chance to undo a death, and her spouse is afraid of what it would bring back."
},
{
"chunk": 24,
"story": 1,
"label": "pressed_to_wrong",
"predicament": "A man pressed by his grieving spouse must decide whether to do something he believes is wicked and dangerous."
},
{
"chunk": 25,
"story": 1,
"label": "dread",
"predicament": "A man who dreads what his wish may bring home must stop his spouse from opening the door to it."
},
{
"chunk": 26,
"story": 1,
"label": "dread",
"predicament": "A man must use his last chance to undo what his spouse most longs for, before she lets it in."
},
{
"chunk": 27,
"story": 2,
"label": "after_your_money",
"predicament": "A man who has hoarded his savings all his life falls ill far from anywhere he could spend them."
},
{
"chunk": 28,
"story": 2,
"label": "after_your_money",
"predicament": "A dying miser, surrounded by people hoping to inherit, is determined that nobody will get his money."
},
{
"chunk": 29,
"story": 2,
"label": "pressed_to_wrong",
"predicament": "Someone who spots a colleague quietly scheming to take unclaimed money must decide whether to expose him or join in."
},
{
"chunk": 30,
"story": 2,
"label": "your_scheme",
"predicament": "Two partners convinced a weaker person is secretly holding a fortune want to find it before anyone else learns of it."
},
{
"chunk": 31,
"story": 2,
"label": "cant_prove_it",
"predicament": "A schemer whose secret searches keep failing decides to confront the person he suspects directly."
},
{
"chunk": 32,
"story": 2,
"label": "after_your_money",
"predicament": "Someone suspected of hiding a windfall is pressed with bribes and threats by people who have no right to it."
},
{
"chunk": 33,
"story": 2,
"label": "your_scheme",
"predicament": "Someone forced to share a hidden windfall admits it is now stuck where neither partner can get at it."
},
{
"chunk": 34,
"story": 2,
"label": "your_scheme",
"predicament": "Distrustful partners stare at a fortune they cannot reach without getting caught, and blame each other for it."
},
{
"chunk": 35,
"story": 2,
"label": "your_scheme",
"predicament": "A man wants something a superior owns without revealing why, and tries flattery to get it."
},
{
"chunk": 36,
"story": 2,
"label": "your_scheme",
"predicament": "A buyer who cannot reveal why he wants something is made to pay more by a seller who senses his eagerness."
},
{
"chunk": 37,
"story": 2,
"label": "your_scheme",
"predicament": "A man who wants to cheat his partner out of a shared prize is watched too closely to slip away."
},
{
"chunk": 38,
"story": 2,
"label": "your_scheme",
"predicament": "A man trying to slip away with a shared prize is held up until the partner he meant to cheat catches him."
},
{
"chunk": 39,
"story": 2,
"label": "your_scheme",
"predicament": "A man who schemed to cheat a partner finds the prize gone and realises he may have been cheated himself."
},
{
"chunk": 40,
"story": 2,
"label": "your_scheme",
"predicament": "A schemer who was outwitted must stand and smile at the people who got the better of him."
},
{
"chunk": 41,
"story": 3,
"label": "none",
"predicament": "A man about to marry is needled by a dependent relative who hints at something to his discredit."
},
{
"chunk": 42,
"story": 3,
"label": "none",
"predicament": "A man in desperate debt makes veiled hints to the relative he hopes will pay it."
},
{
"chunk": 43,
"story": 3,
"label": "past_comes_back",
"predicament": "A man is blackmailed by a relative with old letters that could cost him the person he loves."
},
{
"chunk": 44,
"story": 3,
"label": "let_another_take_blame",
"predicament": "A man who has done something terrible in anger must cover for a relative's sudden disappearance."
},
{
"chunk": 45,
"story": 3,
"label": "hiding_from_close",
"predicament": "A man with a guilty secret must keep calm while someone he loves insists on lingering right at the spot."
},
{
"chunk": 46,
"story": 3,
"label": "hiding_from_close",
"predicament": "A man hiding a crime hears it discussed innocently and betrays his feelings with an outburst."
},
{
"chunk": 47,
"story": 3,
"label": "hiding_from_close",
"predicament": "A guilty man tries to hurry someone away from the scene of his crime, and she insists on staying."
},
{
"chunk": 48,
"story": 3,
"label": "hiding_from_close",
"predicament": "A man hiding something learns it may be discovered by an innocent accident, and must not show his fear."
},
{
"chunk": 49,
"story": 3,
"label": "hiding_from_close",
"predicament": "A man must retrieve something from where his secret lies before anyone else goes looking."
},
{
"chunk": 50,
"story": 3,
"label": "hiding_from_close",
"predicament": "A man secretly searching where his crime lies keeps pulling up reminders of it."
},
{
"chunk": 51,
"story": 3,
"label": "hiding_from_close",
"predicament": "A man who has failed to fix things quietly must now involve others while still keeping his secret."
},
{
"chunk": 52,
"story": 3,
"label": "hiding_from_close",
"predicament": "A man hiding something is offered help he cannot accept without being found out."
},
{
"chunk": 53,
"story": 3,
"label": "none",
"predicament": "A man who must face the thing he did alone gathers his nerve while others watch and wonder."
},
{
"chunk": 54,
"story": 3,
"label": "your_scheme",
"predicament": "A man who hid a crime is dragged down by it at the moment he tries to undo its consequences."
},
{
"chunk": 55,
"story": 4,
"label": "peace_at_home",
"predicament": "A woman who has lost something valuable suspects the people around her, and resents her spouse for not caring as much."
},
{
"chunk": 56,
"story": 4,
"label": "cant_prove_it",
"predicament": "Someone who has lost something accuses the most defenceless person nearby, and punishes her before any proof."
},
{
"chunk": 57,
"story": 4,
"label": "peace_at_home",
"predicament": "A man who thinks a household fuss is foolish keeps defending the accused and is scolded for taking her side."
},
{
"chunk": 58,
"story": 4,
"label": "caught_out",
"predicament": "A man who finds a family inquiry ridiculous cannot keep a straight face in front of its self-important leader."
},
{
"chunk": 59,
"story": 4,
"label": "let_another_take_blame",
"predicament": "A man who sees an innocent neighbour blamed for evidence he himself left must decide whether to own up."
},
{
"chunk": 60,
"story": 4,
"label": "caught_out",
"predicament": "A man who knows more than he says can barely hide his amusement from the self-important person running the inquiry."
},
{
"chunk": 61,
"story": 4,
"label": "cant_prove_it",
"predicament": "A self-important man, teased that a neighbour fancies him, resolves to prove his point by trapping her."
},
{
"chunk": 62,
"story": 4,
"label": "pressed_to_wrong",
"predicament": "A man refuses to let his home be used to entrap a friend, so the schemer plans to go behind his back."
},
{
"chunk": 63,
"story": 4,
"label": "unwelcome_in_home",
"predicament": "A man stuck with an unwelcome relative in his home sees a chance to have him disgraced and driven off."
},
{
"chunk": 64,
"story": 4,
"label": "power_over_you",
"predicament": "A man who has trapped his rival in an embarrassing hiding place decides to make the most of it."
},
{
"chunk": 65,
"story": 4,
"label": "caught_out",
"predicament": "A trapped man must listen in silence while someone tells a neighbour a story about him that he cannot deny."
},
{
"chunk": 66,
"story": 4,
"label": "caught_out",
"predicament": "A man caught somewhere he should not be is handed an embarrassing explanation, and has no better one to offer."
},
{
"chunk": 67,
"story": 5,
"label": "past_comes_back",
"predicament": "A man found out in wrongdoing is ordered by his partner to give up everything and leave by morning."
},
{
"chunk": 68,
"story": 5,
"label": "past_comes_back",
"predicament": "A man offered a harsh but fair way out of his disgrace wants a quieter bargain the other will not give."
},
{
"chunk": 69,
"story": 5,
"label": "pressed_to_wrong",
"predicament": "A man facing ruin threatens the only person who can expose him, and must choose whether to go through with it."
},
{
"chunk": 70,
"story": 5,
"label": "none",
"predicament": "A man who has just done something terrible in rage panics and must steady himself when a figure of the law appears."
},
{
"chunk": 71,
"story": 5,
"label": "none",
"predicament": "A man with a crime behind him and a few hours' start must decide what to take and where to run."
},
{
"chunk": 72,
"story": 5,
"label": "none",
"predicament": "A fugitive planning his escape finds that every choice might be the one that gets him caught."
},
{
"chunk": 73,
"story": 5,
"label": "dread",
"predicament": "A man alone with what he has done hears noises in the dark and fears it has come back for him."
},
{
"chunk": 74,
"story": 5,
"label": "let_another_take_blame",
"predicament": "A guilty man realises a stranger's blunder gives him someone else to blame."
},
{
"chunk": 75,
"story": 5,
"label": "let_another_take_blame",
"predicament": "A man framing an innocent person must act the outraged victim in front of the law."
},
{
"chunk": 76,
"story": 5,
"label": "let_another_take_blame",
"predicament": "A man must play the shocked innocent while the person he framed pleads the truth."
},
{
"chunk": 77,
"story": 5,
"label": "let_another_take_blame",
"predicament": "Someone falsely accused points to the real culprit, who must hold his nerve and keep up his story."
},
{
"chunk": 78,
"story": 5,
"label": "your_scheme",
"predicament": "A man who thought he had got away with it learns the one witness who could hang him is still alive."
},
{
"chunk": 79,
"story": 5,
"label": "your_scheme",
"predicament": "A man whose deceit has collapsed can only watch as the truth comes out and he is taken away."
},
{
"chunk": 80,
"story": 6,
"label": "none",
"predicament": "A penniless newcomer forces his way into a place he does not belong, counting on a secret to back him."
},
{
"chunk": 81,
"story": 6,
"label": "past_comes_back",
"predicament": "A man who has built a respectable new life is found by an old associate who threatens to expose his past."
},
{
"chunk": 82,
"story": 6,
"label": "past_comes_back",
"predicament": "A man pays off a blackmailer and learns the price will be half of everything, for good."
},
{
"chunk": 83,
"story": 6,
"label": "past_comes_back",
"predicament": "A reformed man warns a blackmailer not to push him too far, while the blackmailer sets his sights on the man's family."
},
{
"chunk": 84,
"story": 6,
"label": "none",
"predicament": "A man who has seized power over another knows the danger he is in, and guards himself even as he gloats."
},
{
"chunk": 85,
"story": 6,
"label": "unwelcome_in_home",
"predicament": "A household must make polite space for someone who holds them hostage."
},
{
"chunk": 86,
"story": 6,
"label": "unwelcome_in_home",
"predicament": "Someone watches a person she trusts give way at every turn to an intruder, and demands that he stand up for himself."
},
{
"chunk": 87,
"story": 6,
"label": "past_comes_back",
"predicament": "A man who is being blackmailed hears his tormentor threaten someone he loves, and must choose between swallowing it and hitting back."
},
{
"chunk": 88,
"story": 6,
"label": "your_scheme",
"predicament": "A blackmailer's hold depends on his victim's life, and that victim suddenly collapses."
},
{
"chunk": 89,
"story": 6,
"label": "none",
"predicament": "A blackmailer whose victim is dying must grab what he can before his hold disappears."
},
{
"chunk": 90,
"story": 6,
"label": "past_comes_back",
"predicament": "A dying man pressed by his tormentor bargains for time and for provision for someone he loves."
},
{
"chunk": 91,
"story": 6,
"label": "your_scheme",
"predicament": "A man who thinks his helpless victim is at his mercy leans in close to hear the secret."
},
{
"chunk": 92,
"story": 6,
"label": "power_over_you",
"predicament": "A man who pretended to be helpless strikes at the tormentor who threatened his family."
},
{
"chunk": 93,
"story": 6,
"label": "let_another_take_blame",
"predicament": "A man who has done a desperate thing arranges things so that suspicion falls elsewhere."
},
{
"chunk": 94,
"story": 7,
"label": "none",
"predicament": "A craftsman told his work looks like something else quietly fixes it before anyone else notices."
},
{
"chunk": 95,
"story": 7,
"label": "pretence",
"predicament": "An old man looking for a free drink offers a story in exchange."
},
{
"chunk": 96,
"story": 7,
"label": "dread",
"predicament": "People who laughed at a distant danger panic when it seems to have arrived on their doorstep."
},
{
"chunk": 97,
"story": 7,
"label": "dread",
"predicament": "Someone must choose between keeping himself safe behind a barred door and letting in a stranger in danger."
},
{
"chunk": 98,
"story": 7,
"label": "pretence",
"predicament": "A stranger with a thrilling tale of danger is rewarded by a frightened crowd without anyone checking it."
},
{
"chunk": 99,
"story": 7,
"label": "dread",
"predicament": "A community living in fear of an unseen threat ignores the one doubter, because he has a bad name."
},
{
"chunk": 100,
"story": 7,
"label": "cant_prove_it",
"predicament": "A man who has suffered a loss goes to the authorities for help and is brushed off as a nuisance."
},
{
"chunk": 101,
"story": 7,
"label": "none",
"predicament": "A community suffering repeated losses cannot get help, and cannot agree how to protect itself."
},
{
"chunk": 102,
"story": 7,
"label": "pretence",
"predicament": "A man tells a dramatic story of narrow escape to an audience eager to believe it."
},
{
"chunk": 103,
"story": 7,
"label": "pretence",
"predicament": "A man caught in a boastful lie is 'backed up' by a rival, who now has him trapped by it."
},
{
"chunk": 104,
"story": 7,
"label": "none",
"predicament": "People told their fears are nonsense by outsiders decide to take matters into their own hands."
},
{
"chunk": 105,
"story": 7,
"label": "none",
"predicament": "A band that set out to face a danger returns relieved to have found nothing, and is then called to face it for real."
},
{
"chunk": 106,
"story": 7,
"label": "caught_out",
"predicament": "A man with something to hide has an armed crowd force its way into his home."
},
{
"chunk": 107,
"story": 7,
"label": "caught_out",
"predicament": "A man caught with the evidence of his thefts brazens it out and dares his accusers to prove anything."
},
{
"chunk": 108,
"story": 7,
"label": "caught_out",
"predicament": "A man accused of wrongdoing turns his accusers' own exaggerations into his defence."
},
{
"chunk": 109,
"story": 7,
"label": "cant_prove_it",
"predicament": "People who know they were swindled cannot prove it, and must watch the swindler profit."
},
{
"chunk": 110,
"story": 8,
"label": "none",
"predicament": "A man nervously setting out to do something that matters to him is stopped by a friend with questions."
},
{
"chunk": 111,
"story": 8,
"label": "rival",
"predicament": "A man discovers a friend is after the same person he is, on the same day, and each races to get there first."
},
{
"chunk": 112,
"story": 8,
"label": "rival",
"predicament": "Two rivals agree someone must go first, and neither will be the one to wait."
},
{
"chunk": 113,
"story": 8,
"label": "rival",
"predicament": "Two rivals for the same prize trade claims of priority, and each suspects the other of a trick."
},
{
"chunk": 114,
"story": 8,
"label": "your_scheme",
"predicament": "Two rivals agree to settle who goes first by a game, and one slips in a condition the other does not examine."
},
{
"chunk": 115,
"story": 8,
"label": "your_scheme",
"predicament": "A man who tried to tip a contest in his favour loses it through his own oversight."
},
{
"chunk": 116,
"story": 8,
"label": "rival",
"predicament": "A man who lost the right to try first must wait while his rival takes his chance."
},
{
"chunk": 117,
"story": 8,
"label": "rival",
"predicament": "A man finds the rival he lost to has been refused, but can still use the terms of their bargain to keep him out."
},
{
"chunk": 118,
"story": 8,
"label": "rival",
"predicament": "A man bound by a promise he made watches a rival use it to keep him from what he wants."
},
{
"chunk": 119,
"story": 8,
"label": "caught_out",
"predicament": "A man who has been avoiding someone he cares for must explain why, and the only honest answer is embarrassing."
},
{
"chunk": 120,
"story": 8,
"label": "none",
"predicament": "A man bound by a rash promise finds the person he hoped to win is willing to help him get round it."
},
{
"chunk": 121,
"story": 8,
"label": "none",
"predicament": "Someone arranges to accept the suitor she prefers in a way his rival cannot object to."
},
{
"chunk": 122,
"story": 9,
"label": "newcomer",
"predicament": "A man wakes up somewhere strange with no idea how he got there."
},
{
"chunk": 123,
"story": 9,
"label": "newcomer",
"predicament": "A man in an unknown place looks for his one companion and a first clue to how they came to be there."
},
{
"chunk": 124,
"story": 9,
"label": "power_over_you",
"predicament": "A man who has always been obeyed tries to give orders somewhere nobody knows or believes who he is."
},
{
"chunk": 125,
"story": 9,
"label": "your_scheme",
"predicament": "Two people in trouble together try to put on a united front and end up blaming each other."
},
{
"chunk": 126,
"story": 9,
"label": "power_over_you",
"predicament": "Two people who claim to be respected professionals are treated as impostors and forced to do menial work."
},
{
"chunk": 127,
"story": 9,
"label": "power_over_you",
"predicament": "People brought low insist on their old status, and are mocked for it by those around them."
},
{
"chunk": 128,
"story": 9,
"label": "power_over_you",
"predicament": "A humiliated man pretends to submit while he quietly prepares his revenge."
},
{
"chunk": 129,
"story": 9,
"label": "power_over_you",
"predicament": "A man mocked for claiming skill is suddenly needed for it, and names his price."
},
{
"chunk": 130,
"story": 9,
"label": "power_over_you",
"predicament": "A man who has gained the upper hand over those who humiliated him enjoys every moment of it."
},
{
"chunk": 131,
"story": 9,
"label": "power_over_you",
"predicament": "A man who has secretly caused the trouble only he can cure makes the most of it."
},
{
"chunk": 132,
"story": 9,
"label": "none",
"predicament": "A man who swore he would never pay the person he scorned is forced by suffering to haggle with him."
},
{
"chunk": 133,
"story": 9,
"label": "power_over_you",
"predicament": "People who scorned two outsiders now depend on them completely, and pay for the privilege."
},
{
"chunk": 134,
"story": 9,
"label": "power_over_you",
"predicament": "A man who got even with his tormentors leaves them knowing they were had."
},
{
"chunk": 135,
"story": 10,
"label": "strings",
"predicament": "Someone given news that might change their life cannot take it in, while the people around them itch to know."
},
{
"chunk": 136,
"story": 10,
"label": "strings",
"predicament": "Someone who has never had luck is told of a windfall and is afraid to believe it."
},
{
"chunk": 137,
"story": 10,
"label": "strings",
"predicament": "A man sees a relative come into money and resents that luck never comes his way."
},
{
"chunk": 138,
"story": 10,
"label": "after_your_money",
"predicament": "A man flattered by new friends is told they are really after a relative's money."
},
{
"chunk": 139,
"story": 10,
"label": "after_your_money",
"predicament": "A man trying to expose flattering suitors trips over his own family's sensibilities."
},
{
"chunk": 140,
"story": 10,
"label": "rival",
"predicament": "Two rivals courting the same person keep topping each other's compliments while the family grows fed up."
},
{
"chunk": 141,
"story": 10,
"label": "rival",
"predicament": "A slow-talking man losing a contest to a glib rival looks for a trick instead of better words."
},
{
"chunk": 142,
"story": 10,
"label": "rival",
"predicament": "A man hires someone to spread a false rumour that will drive off his rival, and the hireling drives a hard bargain."
},
{
"chunk": 143,
"story": 10,
"label": "your_scheme",
"predicament": "A man courting for money pretends not to care about it and flatters the one he courts by letting her win."
},
{
"chunk": 144,
"story": 10,
"label": "your_scheme",
"predicament": "A schemer who thinks his deal is done walks away gloating too soon."
},
{
"chunk": 145,
"story": 10,
"label": "your_scheme",
"predicament": "A man makes a public show of selfless devotion, sure that it will cost him nothing."
},
{
"chunk": 146,
"story": 10,
"label": "your_scheme",
"predicament": "A schemer whose plan has worked is thanked for his goodness while the ground starts to shift under him."
},
{
"chunk": 147,
"story": 10,
"label": "your_scheme",
"predicament": "A man who faked a disaster to win a prize learns the disaster is real, and he is bound to his promise."
},
{
"chunk": 148,
"story": 11,
"label": "pretence",
"predicament": "A sceptic among believers is pressed to admit whether he has ever seen anything he could not explain."
},
{
"chunk": 149,
"story": 11,
"label": "newcomer",
"predicament": "A young man who arrived to find no one expecting him has to fill empty days in a strange place."
},
{
"chunk": 150,
"story": 11,
"label": "newcomer",
"predicament": "A traveller lost in the dark and the cold must find shelter before night closes in."
},
{
"chunk": 151,
"story": 11,
"label": "dread",
"predicament": "A worn-out traveller must choose between a long walk on and a nearby shelter he has been warned against."
},
{
"chunk": 152,
"story": 11,
"label": "dread",
"predicament": "A stranger taken in by odd hosts who seem keen to keep him wonders whether he has walked into a trap."
},
{
"chunk": 153,
"story": 11,
"label": "dread",
"predicament": "A guest must sit down in the dark with someone he cannot see, on hosts' terms he does not understand."
},
{
"chunk": 154,
"story": 11,
"label": "dread",
"predicament": "A guest left alone with a stranger he cannot see grows afraid, and must decide whether to trust his hosts."
},
{
"chunk": 155,
"story": 11,
"label": "dread",
"predicament": "A frightened man sees what he feared, and learns it is someone who was hurt and ashamed to be seen."
},
{
"chunk": 156,
"story": 11,
"label": "owning_up",
"predicament": "A man who shrank from someone must make amends, and finds they deserve his respect."
}
]
}
tools/story-to-pack/probe/catalogue/labels-ohenry.json
+5040
View File
File diff suppressed because it is too large Load Diff
tools/story-to-pack/probe/catalogue/labels-wharton.json
+1578
View File
File diff suppressed because it is too large Load Diff
tools/story-to-pack/probe/catalogue/out/ohenry-q14-dev1.partial.json
+1356
View File
File diff suppressed because it is too large Load Diff
tools/story-to-pack/probe/catalogue/pilot40.log
+36
View File
@@ -0,0 +1,36 @@
Pilot, qwen3:14b, shortlist k=8, 14B predicaments, O. Henry scenes 0-39. The output file was later deleted;
these figures were copied by hand from the session transcript (score.py output):
situated? precision 38% recall 93% (model calls 34, hand 14)
label right where both situated: 23% of 13
kappa over all labels incl. none: 0.09
embedding-only baseline (nearest entry) on hand-situated scenes: 36%
hand label inside the shortlist offered: 71%
ohenry-q14-dev1.partial: 30 scenes
situated? precision 67% recall 100% (model calls 21, hand 14)
label right where both situated: 14% of 14
kappa over all labels incl. none: 0.23
ohenry-q14-dev1.partial fit>=3: 30 scenes
situated? precision 62% recall 57% (model calls 13, hand 14)
label right where both situated: 0% of 8
kappa over all labels incl. none: 0.09
6 hand=none model=unspoken fit=3 | Someone and the narrator sit in silence, caught between the quiet intimacy of the moment and the unspoken weig
9 hand=home_unseen model=home_unseen fit=2 | Someone longs for connection but is trapped in a life of monotony and shallow conversation.
10 hand=none model=caught_out fit=3 | A man tries to pay for a purchase with a promise of future money, but his dishonesty sparks a violent confront
11 hand=none model=caught_out fit=2 | A man flees from police pursuit, seeking refuge in an unknown vehicle that offers him a chance to escape, but
12 hand=none model=their_secret fit=3 | A man finds himself unexpectedly drawn into a personal conflict he did not ask for, yet must navigate with cou
13 hand=none model=owning_up fit=3 | Someone must confront a man who has wronged a woman he now feels compelled to protect, risking his own safety
14 hand=none model=compromise_for_money fit=2 | A man struggles to reconcile his growing affection for a woman with the rigid expectations of his social circl
16 hand=station model=compromise_for_money fit=2 | A man finds himself trapped in an awkward encounter with a shopgirl whose icy beauty challenges his composure,
17 hand=station model=pretence fit=2 | A man struggling with his pride and social conventions finds himself awkwardly drawn to a shopgirl, determined
18 hand=station model=compromise_for_money fit=2 | A young woman balances between curiosity and caution as she agrees to meet a man she barely knows, hoping to e
19 hand=station model=unspoken fit=3 | A man torn between his growing feelings for a woman he barely knows and his fear of losing control of his life
20 hand=station model=pretence fit=3 | A man insists a woman marry him, claiming his wealth and status make them equal, but she sees through his word
21 hand=station model=strings fit=2 | A man tries to convince a hesitant woman to leave her mundane life and embark on an extravagant, uncertain jou
22 hand=station model=pretence fit=3 | A woman hides her growing feelings for a man while pretending to be uninterested, hoping to keep their connect
23 hand=none model=hiding_from_close fit=3 | Big someone must navigate the rigid codes of his underground world while seeking a way to break free and claim
24 hand=home_unseen model=home_unseen fit=2 | A man struggles to balance his cultural identity and personal life while navigating the expectations of his co
25 hand=home_unseen model=habit fit=3 | A man accustomed to routine is confronted with an unexpected request for companionship, forcing him to reconsi
26 hand=home_unseen model=hiding_from_close fit=3 | A man struggling to reconcile his desire to display his wife publicly with his fear of judgment finds himself
27 hand=home_unseen model=what_they_hid fit=3 | A man struggles to reconcile his pride with the growing awareness of his wife's hidden talents and the shame o
28 hand=home_unseen model=hiding_from_close fit=3 | A man struggles to reconcile his quiet life with the sudden, overwhelming attention his wife receives, threate
29 hand=home_unseen model=peace_at_home fit=3 | A man faces a choice between social obligation and personal pride, seeking to assert his dignity while preserv
tools/story-to-pack/probe/catalogue/recurrence.log
+68
View File
@@ -0,0 +1,68 @@
== ohenry: 97 stories, 839 scenes; rule: >= 5 scenes, >= 4 stories, no story over 50%
scenes labelled: 475 (57%)
PASS pretence 52 scenes 12 stories top story 15%
PASS benefactor 48 scenes 14 stories top story 17%
PASS home_unseen 47 scenes 11 stories top story 15%
PASS station 47 scenes 10 stories top story 17%
PASS compromise_for_money 38 scenes 13 stories top story 18%
PASS newcomer 32 scenes 8 stories top story 22%
PASS sure_thing 30 scenes 7 stories top story 23%
PASS sacrifice 25 scenes 8 stories top story 28%
PASS rival 24 scenes 6 stories top story 29%
PASS despised 20 scenes 7 stories top story 30%
PASS pressed_to_wrong 19 scenes 5 stories top story 32%
PASS wrongly_judged 19 scenes 7 stories top story 26%
PASS sign 17 scenes 5 stories top story 35%
strings 15 scenes 4 stories top story 53%
PASS unspoken 13 scenes 4 stories top story 38%
overlooked 9 scenes 3 stories top story 67%
PASS homecoming 8 scenes 4 stories top story 50%
PASS patron 8 scenes 4 stories top story 50%
habit 4 scenes 3 stories top story 50%
-> 16 recurring situations of 19 present
== wharton: 10 stories, 262 scenes; rule: >= 5 scenes, >= 3 stories, no story over 50%
scenes labelled: 129 (49%)
PASS caught_out 12 scenes 4 stories top story 50%
PASS compromise_for_money 12 scenes 4 stories top story 50%
PASS past_comes_back 12 scenes 3 stories top story 50%
PASS what_they_hid 10 scenes 6 stories top story 20%
PASS peace_at_home 8 scenes 5 stories top story 38%
PASS newcomer 8 scenes 3 stories top story 50%
PASS rival 8 scenes 4 stories top story 50%
PASS power_over_you 8 scenes 4 stories top story 50%
PASS sacrifice 8 scenes 5 stories top story 38%
PASS owning_up 8 scenes 4 stories top story 38%
PASS home_unseen 7 scenes 4 stories top story 43%
PASS go_between 6 scenes 3 stories top story 50%
PASS waiting 6 scenes 3 stories top story 50%
PASS own_words_back 6 scenes 3 stories top story 50%
PASS strings 5 scenes 4 stories top story 40%
PASS their_secret 5 scenes 3 stories top story 40%
-> 16 recurring situations of 16 present
== jacobs: 12 stories, 157 scenes; rule: >= 5 scenes, >= 3 stories, no story over 50%
scenes labelled: 137 (87%)
PASS your_scheme 23 scenes 8 stories top story 39%
PASS dread 12 scenes 4 stories top story 42%
rival 11 scenes 3 stories top story 55%
power_over_you 11 scenes 3 stories top story 82%
hiding_from_close 10 scenes 2 stories top story 80%
PASS caught_out 10 scenes 5 stories top story 40%
PASS let_another_take_blame 8 scenes 5 stories top story 50%
strings 8 scenes 2 stories top story 62%
past_comes_back 8 scenes 3 stories top story 62%
peace_at_home 5 scenes 2 stories top story 60%
PASS pressed_to_wrong 5 scenes 4 stories top story 40%
after_your_money 5 scenes 2 stories top story 60%
PASS cant_prove_it 5 scenes 3 stories top story 40%
pretence 5 scenes 2 stories top story 80%
unwelcome_in_home 4 scenes 3 stories top story 50%
newcomer 4 scenes 2 stories top story 50%
owning_up 3 scenes 2 stories top story 67%
-> 6 recurring situations of 17 present
== catalogue coverage: entries found in 3 / 2 / 1 corpora
3: rival, strings, newcomer
2: compromise_for_money, past_comes_back, pretence, caught_out, home_unseen, peace_at_home, power_over_you, pressed_to_wrong, sacrifice, owning_up
1: sure_thing, your_scheme, after_your_money, hiding_from_close, their_secret, what_they_hid, wrongly_judged, let_another_take_blame, cant_prove_it, station, unwelcome_in_home, benefactor, patron, go_between, own_words_back, homecoming, dread, waiting, unspoken, overlooked, sign, despised, habit
tools/story-to-pack/probe/catalogue/score.py
+70
View File
@@ -0,0 +1,70 @@
"""Score classifier output against the hand labels. No inference.
python3 score.py out/ohenry-q14.json [more files] # or --selftest
Reports: (1) does it find the situated scenes (label vs none); (2) on scenes both call situated, is the
label right; (3) Cohen's kappa over everything; (4) the embedding-only baseline (nearest entry, no
model) on the same scenes; (5) the recurring situations a pack would get under the one shared rule,
from the model's labels against the hand labels. The recall of the K-nearest shortlist is reported
too: a hand label that was never offered cannot be chosen.
"""
import collections, json, pathlib, random, sys
HERE = pathlib.Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
from catalogue_src import recurs, load_corpus
def kappa(a, b):
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
ca, cb = collections.Counter(a), collections.Counter(b)
pe = sum(ca[k] * cb[k] for k in ca) / n / n
return (po - pe) / (1 - pe) if pe < 1 else 1.0
def score(corpus, hand, pred, nearest=None, cands=None, name=''):
chunks = sorted(pred)
h = [hand[c] for c in chunks]; p = [pred[c] for c in chunks]
tp = sum(x != 'none' and y != 'none' for x, y in zip(h, p))
prec = tp / max(1, sum(y != 'none' for y in p)); rec = tp / max(1, sum(x != 'none' for x in h))
both = [(x, y) for x, y in zip(h, p) if x != 'none' and y != 'none']
acc = sum(x == y for x, y in both) / max(1, len(both))
lines = [f'{name}: {len(chunks)} scenes',
f' situated? precision {prec:.0%} recall {rec:.0%} (model calls {sum(y != "none" for y in p)}, hand {sum(x != "none" for x in h)})',
f' label right where both situated: {acc:.0%} of {len(both)}',
f' kappa over all labels incl. none: {kappa(h, p):.2f}']
if nearest:
sit = [c for c in chunks if hand[c] != 'none']
lines.append(f' embedding-only baseline (nearest entry) on hand-situated scenes: {sum(nearest[c] == hand[c] for c in sit) / max(1, len(sit)):.0%}')
lines.append(f' hand label inside the shortlist offered: {sum(hand[c] in cands[c] for c in sit) / max(1, len(sit)):.0%}')
chunks_all, _, _ = load_corpus(corpus); story = [c['story'] for c in chunks_all]
def passing(lab):
mem = collections.defaultdict(list)
for c, l in lab.items():
if l != 'none': mem[l].append(c)
return {l: m for l, m in mem.items() if recurs(m, story)[0]}
ph = passing({c: hand[c] for c in chunks}); pp = passing({c: pred[c] for c in chunks})
common = set(ph) & set(pp)
lines.append(f' recurring situations: hand {len(ph)}, model {len(pp)}, both {len(common)}')
lines.append(f' model only: {", ".join(sorted(set(pp) - set(ph))) or "-"}')
lines.append(f' hand only: {", ".join(sorted(set(ph) - set(pp))) or "-"}')
for l in sorted(common, key=lambda l: -len(ph[l])):
a, b = set(ph[l]), set(pp[l])
lines.append(f' {l:24s} hand {len(a):3d} model {len(b):3d} overlap {len(a & b):3d} (model precision {len(a & b) / len(b):.0%}, recall {len(a & b) / len(a):.0%})')
return '\n'.join(lines)
if __name__ == '__main__':
out = []
if sys.argv[1:] == ['--selftest']:
lab = json.load(open(HERE / 'labels-ohenry.json'))['labels']; hand = {x['chunk']: x['label'] for x in lab}
out.append(score('ohenry', hand, dict(hand), name='SELFTEST hand vs itself (expect 100% / kappa 1.00)'))
rnd = random.Random(1); ids = sorted(set(hand.values()))
out.append(score('ohenry', hand, {c: rnd.choice(ids) for c in hand}, name='SELFTEST random labels (expect kappa ~0)'))
for f in sys.argv[1:]:
if f.startswith('--'): continue
corpus = pathlib.Path(f).stem.split('-')[0]
lab = json.load(open(HERE / f'labels-{corpus}.json'))['labels']; hand = {x['chunk']: x['label'] for x in lab}
res = json.load(open(f))['results']
minfit = int(next((a.split('=')[1] for a in sys.argv if a.startswith('--minfit=')), 0))
rng = next((a.split('=')[1] for a in sys.argv if a.startswith('--range=')), None)
if rng: lo, hi = map(int, rng.split(':')); res = [r for r in res if lo <= r['chunk'] < hi]
pred = {r['chunk']: (r['label'] if (r.get('fit') or 3) >= minfit else 'none') for r in res}
out.append(score(corpus, hand, pred, {r['chunk']: r['nearest'] for r in res}, {r['chunk']: r['candidates'] for r in res}, name=pathlib.Path(f).stem + (f' fit>={minfit}' if minfit else '') + (f' scenes {rng}' if rng else '')))
print('\n\n'.join(out))