JesseMarkowitzandClaude Opus 5
e5617b86ba
The pipeline's embed-and-cluster step is dead, and this commit holds both the evidence for that and the step proposed to replace it. Predicaments. Scenes are re-described as "what the person is up against", with no names, jobs or places, then embedded and clustered (redescribe.py, topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading both side by side. Two defects the pilot exposed are fixed: split.py missed titles in quotes and a contents subtitle after a dash, so three stories had been merged into their neighbours, and strip_names.py read New York place names as people. The corrected corpus is probe/v2 (97 stories, 839 scenes); carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from 20% to 13% at k=60, short of the pre-registered 10%. Hand references. Three corpora were read scene by scene and written up by hand, under the same prompt rules the local models get, as a baseline to judge them against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a readable page. No inference was used for any of them. Catalogue. probe/catalogue maps every hand group in the three references onto 36 situation entries, with an answer key per corpus and one recurrence rule applied to all three. classify.py assigns a scene one entry or none, leave-one-corpus- out; score.py checks it against the key, with a self-test on random labels. Why clustering is out: hand-written predicaments, embedded and clustered exactly as the model's were, agree with the hand grouping at ARI 0.05 — no better than the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even shortlist: the hand label is the nearest entry 13% of the time and in the top 8 half the time. The classification runs are not here. The dev and test runs are pre-registered in probe/catalogue/README.md with the bar set beforehand, and are blocked on the inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene partial output in out/ is not a result. Review page. The situation review is now a browser page rather than JSON edited by hand (review_page.py, review_page_logic.cjs with Node tests, format schema v2). It has never been rendered in a real browser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
78 lines
3.1 KiB
Python
78 lines
3.1 KiB
Python
"""Check a saved review and turn it into situations.json.
|
|
|
|
python3 apply_review.py review/k60-s24 # finds the saved review itself
|
|
python3 apply_review.py review/k60-s24 ~/Downloads/review-k60-s24.json
|
|
|
|
Without a file named, it uses the newest review-<run>*.json in the run directory, and
|
|
otherwise the newest one in ~/Downloads, where a browser saves it. It prints every problem
|
|
at once and writes nothing until the review passes; then it writes situations.json into the
|
|
run directory.
|
|
"""
|
|
import datetime, json, os, pathlib, sys
|
|
import review_format as rf
|
|
|
|
|
|
def downloads_dir():
|
|
return pathlib.Path(os.environ.get('STP_DOWNLOADS', pathlib.Path.home() / 'Downloads'))
|
|
|
|
|
|
def find_review(run_dir, run_id):
|
|
for folder in (run_dir, downloads_dir()):
|
|
if folder.is_dir():
|
|
found = sorted(folder.glob(f'review-{run_id}*.json'), key=lambda p: p.stat().st_mtime, reverse=True)
|
|
if found:
|
|
return found[0]
|
|
return None
|
|
|
|
|
|
def main(argv):
|
|
if len(argv) not in (2, 3):
|
|
print(__doc__.strip())
|
|
return 2
|
|
run_dir = pathlib.Path(argv[1])
|
|
cand_path = run_dir / 'candidates.json'
|
|
if not cand_path.exists():
|
|
print(f'{run_dir} has no candidates.json. Check the folder name, or run review.py first.')
|
|
return 1
|
|
candidates = json.loads(cand_path.read_text(encoding='utf-8'))
|
|
run_id = candidates['run']['id']
|
|
|
|
review_path = pathlib.Path(argv[2]).expanduser() if len(argv) == 3 else find_review(run_dir, run_id)
|
|
if review_path is None or not review_path.exists():
|
|
print(f'No saved review found for run {run_id}.')
|
|
print(f'Open {run_dir / "REVIEW.html"} in a browser, and at the end press "Save review file".')
|
|
print(f'It saves review-{run_id}.json into Downloads, where this looks for it.')
|
|
return 1
|
|
print(f'reading {review_path}')
|
|
try:
|
|
form = json.loads(review_path.read_text(encoding='utf-8'))
|
|
except json.JSONDecodeError:
|
|
print('That file is damaged and cannot be read. Save the review again from REVIEW.html; '
|
|
'your progress is still in the browser.')
|
|
return 1
|
|
|
|
errors, warnings, result = rf.validate(candidates, form)
|
|
for w in warnings:
|
|
print(f'note: {w}')
|
|
if errors:
|
|
print(f'\nThe review is not finished. {len(errors)} thing(s) to sort out in REVIEW.html, '
|
|
f'then save the review file again:')
|
|
for e in errors:
|
|
print(f' - {e}')
|
|
return 1
|
|
|
|
result['applied'] = datetime.datetime.now().astimezone().isoformat(timespec='seconds')
|
|
result['review_file'] = str(review_path)
|
|
out = run_dir / 'situations.json'
|
|
out.write_text(json.dumps(result, indent=1, ensure_ascii=False), encoding='utf-8')
|
|
c = result['counts']
|
|
print(f'\n{out}: {c["situations"]} situations ({c["kept"]} kept, {c["same"]} merged in, '
|
|
f'{c["dropped"]} dropped, {c["left_out"]} extras left out).')
|
|
for s in result['situations']:
|
|
print(f' {s["id"]} {s["situation"]} ({s["size"]} scenes, {s["stories"]} stories)')
|
|
return 0
|
|
|
|
|
|
if __name__ == '__main__':
|
|
sys.exit(main(sys.argv))
|