Jesse.Markowitz/TheLadder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace clustering with a catalogue, and hand-write the references to judge it against

Browse Source 
The pipeline's embed-and-cluster step is dead, and this commit holds both the
evidence for that and the step proposed to replace it.

Predicaments. Scenes are re-described as "what the person is up against", with
no names, jobs or places, then embedded and clustered (redescribe.py,
topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading
both side by side. Two defects the pilot exposed are fixed: split.py missed
titles in quotes and a contents subtitle after a dash, so three stories had been
merged into their neighbours, and strip_names.py read New York place names as
people. The corrected corpus is probe/v2 (97 stories, 839 scenes);
carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from
20% to 13% at k=60, short of the pre-registered 10%.

Hand references. Three corpora were read scene by scene and written up by hand,
under the same prompt rules the local models get, as a baseline to judge them
against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's
Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of
the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a
readable page. No inference was used for any of them.

Catalogue. probe/catalogue maps every hand group in the three references onto 36
situation entries, with an answer key per corpus and one recurrence rule applied
to all three. classify.py assigns a scene one entry or none, leave-one-corpus-
out; score.py checks it against the key, with a self-test on random labels.

Why clustering is out: hand-written predicaments, embedded and clustered exactly
as the model's were, agree with the hand grouping at ARI 0.05 — no better than
the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even
shortlist: the hand label is the nearest entry 13% of the time and in the top 8
half the time.

The classification runs are not here. The dev and test runs are pre-registered
in probe/catalogue/README.md with the bar set beforehand, and are blocked on the
inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene
partial output in out/ is not a result.

Review page. The situation review is now a browser page rather than JSON edited
by hand (review_page.py, review_page_logic.cjs with Node tests, format schema
v2). It has never been rendered in a real browser.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
This commit is contained in:
JesseMarkowitz
2026-09-20 17:19:42 -04:00
co-authored by Claude Opus 5
parent fa3769d0fe
commit e5617b86ba
135 changed files with 82027 additions and 1778 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/story-to-pack/probe/review_format.py
+91 -105
View File
@@ -1,51 +1,46 @@
"""The human review form: its schema, member ids, checks, and conversion to situations.json.
"""The saved review: its format, its checks, and conversion to situations.json.
Kept free of clustering code so apply_review.py and the tests load instantly.
Kept free of clustering and page code so apply_review.py and the tests load instantly.
A review run has three files in one directory:
candidates.json written by review.py: what the program found (read-only)
review.json written blank by review.py, filled in by the reviewer
situations.json written by apply_review.py once review.json passes every check
A review run directory holds:
candidates.json written by review.py: the groups the program found (read-only)
REVIEW.html written by review.py: the page the reviewer works in
review-<run>.json saved from that page, usually into Downloads
situations.json written by apply_review.py once the review passes every check
The page (review_page_logic.cjs) applies the same rules as it goes, so a reviewer sees a
problem where it happens. This module is still the authority: apply_review.py checks the
saved file again before anything downstream reads it.
"""
import collections
SCHEMA_VERSION = 1
SCHEMA_VERSION = 2
MIN_MEMBERS, MIN_STORIES, MAX_DOMINANT = 5, 4, 0.40
MAX_SITUATION_CHARS = 200
MAX_FIELD_CHARS = 80
DECISIONS = {
'accepted': ('accept', 'reject', 'merge'), # passed the gate: the reviewer must decide
'borderline': ('skip', 'accept', 'merge'), # narrowly failed: optional, "skip" by default
'accepted': ('keep', 'drop', 'same'), # passed the gate: a decision is required
'borderline': ('skip', 'keep', 'same'), # nearly passed: optional, left out by default
}
RULES = {'minMembers': MIN_MEMBERS, 'minStories': MIN_STORIES, 'maxDominant': MAX_DOMINANT,
'maxFieldChars': MAX_FIELD_CHARS}
def member_id(cid, n):
"""C04.07 — the 7th scene of candidate C04."""
"""C04.07 — the 7th scene of group C04."""
return f'{cid}.{n:02d}'
def blank_entry(cid, kind):
return {
'id': cid,
'kind': kind,
'decision': None if kind == 'accepted' else 'skip',
'situation': '',
'actor': '',
'counterpart': '',
'stakes': '',
'exclude': [],
'merge_into': None,
'notes': '',
}
return {'id': cid, 'decision': None if kind == 'accepted' else 'skip', 'same_as': None,
'actor': '', 'wants': '', 'counterpart': '', 'excluded': []}
def blank_form(candidates):
return {
'schema_version': SCHEMA_VERSION,
'run': candidates['run']['id'],
'instructions': 'Read REVIEW.md in this directory. Fill in each entry below, then run: '
f'python3 apply_review.py review/{candidates["run"]["id"]}',
'entries': [blank_entry(cid, c['kind']) for cid, c in candidates['candidates'].items()],
}
def display_name(candidates, cid):
"""What the page calls a group: "Group 3", or "Extra group 2" for a near miss."""
kind = candidates['candidates'][cid]['kind']
same_kind = [c for c, v in candidates['candidates'].items() if v['kind'] == kind]
n = same_kind.index(cid) + 1
return f'Group {n}' if kind == 'accepted' else f'Extra group {n}'
def shape(members):
@@ -54,6 +49,10 @@ def shape(members):
return len(members), len(counts), (max(counts.values()) / len(members) if members else 0.0)
def sentence(entry):
return f'A {entry["actor"].strip()} wants {entry["wants"].strip()} from {entry["counterpart"].strip()}.'
def _text(value):
return value.strip() if isinstance(value, str) else ''
@@ -61,104 +60,93 @@ def _text(value):
def validate(candidates, form):
"""Return (errors, warnings, situations). situations is None whenever there is any error.
Every problem is collected, so the reviewer can fix them all in one pass.
Every problem is collected, so the reviewer can fix them all in one pass. Messages name
groups the way the page does, and say what to do there.
"""
errors, warnings = [], []
cands = candidates['candidates']
run = candidates['run']
name = lambda cid: display_name(candidates, cid)
if form.get('schema_version') != SCHEMA_VERSION:
errors.append(f'"schema_version" is {form.get("schema_version")!r}; this checker reads {SCHEMA_VERSION}.')
if form.get('run') != candidates['run']['id']:
errors.append(f'This form is for run {form.get("run")!r}, but candidates.json is run '
f'{candidates["run"]["id"]!r}. Use the review.json from the same directory.')
errors.append(f'This review file was made by a different version of the review tool '
f'(format {form.get("schema_version")!r}; this checker reads {SCHEMA_VERSION}). '
f'Regenerate the page with review.py and review in that.')
return errors, warnings, None
if form.get('run') != run['id']:
errors.append(f'This review file is for run {form.get("run")!r}, but this directory is run '
f'{run["id"]!r}. Use the file saved from this run\'s REVIEW.html.')
if run.get('fingerprint') and form.get('fingerprint') and form['fingerprint'] != run['fingerprint']:
errors.append('This review file was made for a different set of groups that used the same run '
'name. Review again in the current REVIEW.html.')
entries = form.get('entries')
if not isinstance(entries, list):
errors.append('"entries" must be a list of candidate entries.')
errors.append('The review file has no list of groups in it. Save it again from REVIEW.html.')
return errors, warnings, None
by_id = {}
for e in entries:
cid = e.get('id') if isinstance(e, dict) else None
if cid not in cands:
errors.append(f'An entry has id {cid!r}, which is not a candidate in this run.')
elif cid in by_id:
errors.append(f'{cid}: appears more than once; keep one entry.')
else:
errors.append(f'The review file mentions a group {cid!r} that is not in this run.')
elif cid not in by_id:
by_id[cid] = e
for cid in cands:
if cid not in by_id:
errors.append(f'{cid}: its entry is missing from the form. Restore it (the blank entry is '
f'{{"id": "{cid}", ...}}) and set "decision".')
errors.append(f'{name(cid)} is missing from the review file. Save it again from REVIEW.html.')
for cid, e in by_id.items():
kind = cands[cid]['kind']
allowed = DECISIONS[kind]
decision = e.get('decision')
if decision is None:
errors.append(f'{cid}: no decision yet. Set "decision" to one of: {", ".join(allowed)}.')
errors.append(f'{name(cid)}: not decided yet. Open it in REVIEW.html and choose keep, drop, '
f'or same as another group.')
continue
if decision not in allowed:
errors.append(f'{cid}: "decision" is {decision!r}; for a {kind} candidate use one of: '
f'{", ".join(allowed)}.')
if decision not in DECISIONS[kind]:
errors.append(f'{name(cid)}: the decision {decision!r} is not one this group can have.')
continue
members = cands[cid]['members']
valid = {m['member'] for m in members}
exclude = e.get('exclude', [])
if not isinstance(exclude, list):
errors.append(f'{cid}: "exclude" must be a list of member ids, like ["{cid}.03"].')
exclude = []
unknown = [x for x in exclude if x not in valid]
if unknown:
errors.append(f'{cid}: "exclude" lists {", ".join(map(str, unknown))}, which are not scenes of '
f'{cid}; its scenes are {member_id(cid, 1)} to {member_id(cid, len(members))}.')
excluded = e.get('excluded', [])
if not isinstance(excluded, list) or any(x not in valid for x in excluded):
errors.append(f'{name(cid)}: the unticked scenes do not match this group. Save the review '
f'again from REVIEW.html.')
if decision == 'accept':
for field in ('situation', 'actor', 'counterpart'):
if not _text(e.get(field)):
errors.append(f'{cid}: accepted, so "{field}" must be filled in.')
situation = _text(e.get('situation'))
if len(situation) > MAX_SITUATION_CHARS:
errors.append(f'{cid}: "situation" is {len(situation)} characters; keep it to one sentence '
f'of at most {MAX_SITUATION_CHARS}.')
if situation and 'want' not in situation.lower():
warnings.append(f'{cid}: "situation" does not say who wants what ("… wants … from …"). '
f'Accepted as written.')
if not _text(e.get('stakes')):
warnings.append(f'{cid}: "stakes" is empty.')
if decision == 'keep':
missing = [label for key, label in (('actor', 'who wants something'), ('wants', 'what they want'),
('counterpart', 'who they want it from')) if not _text(e.get(key))]
if missing:
errors.append(f'{name(cid)}: kept, but its name is not finished. Fill in '
f'{", ".join(missing)} in "A ___ wants ___ from ___".')
for key in ('actor', 'wants', 'counterpart'):
if len(_text(e.get(key))) > MAX_FIELD_CHARS:
errors.append(f'{name(cid)}: "{key}" is longer than {MAX_FIELD_CHARS} characters.')
if isinstance(excluded, list) and len(excluded) > len(members) / 2:
warnings.append(f'{name(cid)}: more than half its scenes are unticked. Check the name '
f'describes what the group is mostly about.')
if decision == 'merge':
target = e.get('merge_into')
if target == cid:
errors.append(f'{cid}: cannot merge into itself.')
elif target not in by_id:
errors.append(f'{cid}: "merge_into" is {target!r}, which is not a candidate id in this form.')
elif by_id[target].get('decision') != 'accept':
errors.append(f'{cid}: merges into {target}, but {target} has decision '
f'{by_id[target].get("decision")!r}. Merge only into an accepted candidate, '
f'or accept {cid} instead.')
elif e.get('merge_into') not in (None, ''):
warnings.append(f'{cid}: "merge_into" is set but the decision is {decision!r}; it is ignored.')
if decision in ('reject', 'skip') and exclude:
warnings.append(f'{cid}: "exclude" is ignored because the decision is {decision!r}.')
if decision in ('accept', 'merge') and len(exclude) > len(members) / 2:
warnings.append(f'{cid}: more than half of its scenes are excluded. Check that the situation '
f'named is what this candidate is mostly about.')
if decision == 'same':
target = e.get('same_as')
if target == cid or target not in by_id:
errors.append(f'{name(cid)}: marked "same as" a group that is not in this run.')
elif by_id[target].get('decision') != 'keep':
errors.append(f'{name(cid)}: marked the same as {name(target)}, but {name(target)} is not '
f'kept. Keep {name(target)}, or choose differently for {name(cid)}.')
if errors:
return errors, warnings, None
groups = collections.OrderedDict((cid, [cid]) for cid, e in by_id.items() if e['decision'] == 'accept')
groups = collections.OrderedDict((cid, [cid]) for cid, e in by_id.items() if e['decision'] == 'keep')
for cid, e in by_id.items():
if e['decision'] == 'merge':
groups[e['merge_into']].append(cid)
if e['decision'] == 'same':
groups[e['same_as']].append(cid)
situations = []
for cid, sources in groups.items():
members, seen = [], set()
for src in sources:
excluded = set(by_id[src].get('exclude', []))
excluded = set(by_id[src].get('excluded', []))
for m in cands[src]['members']:
if m['member'] in excluded or m['chunk'] in seen:
continue
@@ -166,22 +154,20 @@ def validate(candidates, form):
members.append(m)
size, stories, dominant = shape(members)
if size < MIN_MEMBERS or stories < MIN_STORIES or dominant > MAX_DOMINANT:
merged = f' (with {", ".join(sources[1:])} merged in)' if len(sources) > 1 else ''
errors.append(f'{cid}{merged}: after exclusions {size} scenes remain from {stories} stories, the '
f'largest story holding {dominant:.0%}. A situation needs at least {MIN_MEMBERS} '
f'scenes from at least {MIN_STORIES} stories, with no story above '
f'{MAX_DOMINANT:.0%}. Exclude fewer scenes, merge another candidate into it, '
f'or reject it.')
merged = f' (with {", ".join(name(s) for s in sources[1:])} merged in)' if len(sources) > 1 else ''
errors.append(f'{name(cid)}{merged}: {size} ticked scenes from {stories} stories, the largest '
f'story holding {dominant:.0%}. A situation needs at least {MIN_MEMBERS} scenes '
f'from at least {MIN_STORIES} stories, with no story above {MAX_DOMINANT:.0%}. '
f'Tick more scenes, mark another group the same as it, or drop it.')
continue
e = by_id[cid]
situations.append({
'id': f'S{len(situations) + 1:02d}',
'from': sources,
'situation': _text(e['situation']),
'situation': sentence(e),
'actor': _text(e['actor']),
'wants': _text(e['wants']),
'counterpart': _text(e['counterpart']),
'stakes': _text(e.get('stakes')),
'notes': [_text(by_id[s].get('notes')) for s in sources if _text(by_id[s].get('notes'))],
'size': size,
'stories': stories,
'members': [{k: m[k] for k in ('member', 'chunk', 'story', 'title', 'summary')} for m in members],
@@ -192,11 +178,11 @@ def validate(candidates, form):
decided = collections.Counter(e['decision'] for e in by_id.values())
result = {
'schema_version': SCHEMA_VERSION,
'run': candidates['run'],
'counts': {'candidates': len(cands), 'situations': len(situations), 'accepted': decided['accept'],
'merged': decided['merge'], 'rejected': decided['reject'], 'skipped': decided['skip']},
'run': run,
'counts': {'groups': len(cands), 'situations': len(situations), 'kept': decided['keep'],
'same': decided['same'], 'dropped': decided['drop'], 'left_out': decided['skip']},
'situations': situations,
'rejected': [cid for cid, e in by_id.items() if e['decision'] == 'reject'],
'skipped': [cid for cid, e in by_id.items() if e['decision'] == 'skip'],
'dropped': [cid for cid, e in by_id.items() if e['decision'] == 'drop'],
'left_out': [cid for cid, e in by_id.items() if e['decision'] == 'skip'],
}
return errors, warnings, result