JesseMarkowitzandClaude Opus 5
e5617b86ba
The pipeline's embed-and-cluster step is dead, and this commit holds both the evidence for that and the step proposed to replace it. Predicaments. Scenes are re-described as "what the person is up against", with no names, jobs or places, then embedded and clustered (redescribe.py, topic_words.py, topic_share.py). The pilot chose qwen3:14b over 3b by reading both side by side. Two defects the pilot exposed are fixed: split.py missed titles in quotes and a contents subtitle after a dash, so three stories had been merged into their neighbours, and strip_names.py read New York place names as people. The corrected corpus is probe/v2 (97 stories, 839 scenes); carry_summaries.py reuses the 829 unchanged v1 summaries. Topic share fell from 20% to 13% at k=60, short of the pre-registered 10%. Hand references. Three corpora were read scene by scene and written up by hand, under the same prompt rules the local models get, as a baseline to judge them against: O. Henry (probe/v2/claude, 839 scenes, 20 situations), Wharton's Descent of Man (probe/wharton, 262 scenes, 16 groups) and Jacobs's The Lady of the Barge (probe/jacobs, 157 scenes, 19 groups). Each has its own README and a readable page. No inference was used for any of them. Catalogue. probe/catalogue maps every hand group in the three references onto 36 situation entries, with an answer key per corpus and one recurrence rule applied to all three. classify.py assigns a scene one entry or none, leave-one-corpus- out; score.py checks it against the key, with a self-test on random labels. Why clustering is out: hand-written predicaments, embedded and clustered exactly as the model's were, agree with the hand grouping at ARI 0.05 — no better than the 14B text's 0.07. Better rewriting cannot rescue it. Embeddings cannot even shortlist: the hand label is the nearest entry 13% of the time and in the top 8 half the time. The classification runs are not here. The dev and test runs are pre-registered in probe/catalogue/README.md with the bar set beforehand, and are blocked on the inference host, whose GPU has fallen off the PCIe bus three times. The 30-scene partial output in out/ is not a result. Review page. The situation review is now a browser page rather than JSON edited by hand (review_page.py, review_page_logic.cjs with Node tests, format schema v2). It has never been rendered in a real browser. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014BygvsUXV9eU6oHkTCkKZ1
story-to-pack
Research toward a tool that builds a The Ladder content pack from a corpus of stories.
Independent of the game. Nothing in src/, content/ or test/ imports anything here, and
nothing here is part of npm test or npm run build. It lives on the story-to-pack branch so it is
versioned alongside the pack format it targets, without touching main.
- What was decided and why:
DESIGN.md - How situations are selected, and every test of it:
SELECTION.md - The 2026-09-10 scripts, recovered from a transcript:
recovered/
Status
Selection was settled as gate G2 followed by a structured human review (see below). Every automated alternative tried — three more geometric gates and two model-judged gates — was worse on fresh, blind-labelled data. The first real review then found G2's groups were topics (artists, police, hotels), not situations. Now under test: re-describe every scene as its main person's predicament with qwen3:14b, regroup, and read the groups (SELECTION.md, "Predicaments"). Nothing downstream of selection (paths and directions, numbers, pack output) is built yet.
Layout
probe/ holds the v1 data every G1–G4 result was measured on. probe/v2/ is the corrected corpus
(97 stories, 839 scenes: v1's splitter merged three quoted-title stories into their neighbours), built by
running the same scripts from inside v2/ (python3 ../split.py, and so on).
probe/
corpus/ O. Henry's Four Million cycle, Project Gutenberg (public domain)
split.py, chunk.py volumes -> stories.json -> chunks.json (838 scenes in v1, 839 in v2)
carry_summaries.py v1 summaries onto v2 scenes; only new scene text goes to the model
redescribe.py scene -> its main person's predicament, via the model -> predicaments.json
topic_words.py, topic_share.py how much groups are held together by a job, relationship or place word
pilot_page.py side-by-side page for the 3B / 14B predicament pilot -> pilot-compare.html
groups_page.py reading page for regrouped predicaments -> v2/groups-*.html
ollama.py the only module that talks to an inference host (host from STP_OLLAMA)
summarise.py scene -> one sentence, via the model -> summaries.json
strip_names.py names removed deterministically -> summaries_clean.json
embed.py texts -> vectors -> embeddings.json, summary_embeddings.json
measure.py, signals.py, gate2.py, gate3.py, judge.py … the selection experiments
labels-*.json, sheet-*.txt, candidates-*.json, *-map.json blind labels and what they labelled
review.py G2 on a partition -> REVIEW.html and candidates.json
review_page.py renders the page; picks out role words for each group
review_page.html the page template
review_page_logic.cjs the rules the page applies (also run by the Node tests)
review_format.py the saved review's format, checks and conversion
apply_review.py saved review -> situations.json
test_review.py, test_review_page.mjs tests
review/<run>/ REVIEW.html, candidates.json, and situations.json once applied
Not committed (see .gitignore): the inference host's power and kernel logs, and caches that the
scripts rebuild on first use (sim.pkl, null*.pkl, coassoc.json). The embeddings are committed
(about 16 MB) because recreating them needs the inference host.
Running
Everything runs from probe/ with Python 3 and no third-party packages.
Anything that calls a model reads the host from the environment and never from a file:
export STP_OLLAMA=http://<inference-host>:11434
Reviewing situations
The review happens in a web page. Nobody edits JSON.
cd probe
python3 review.py # k = 60, seed 24 -> review/k60-s24/REVIEW.html
-
Open
review/k60-s24/REVIEW.htmlin a browser (double-click it; nothing to install). The first screen explains the job with examples. -
One group per screen. For each: keep it, drop it, or mark it the same as a group already kept; untick scenes that don't fit; and fill in "A ___ wants ___ from ___", with the people named in its scenes offered as click-to-fill words. The page shows as you go whether a group still has enough scenes, and colours each group in the strip across the top. Keys: K keep, D drop, ← →.
-
Progress saves in the browser after every change, so you can close it and come back.
-
"Finish & save" lists anything unfinished and saves
review-k60-s24.json, usually to Downloads. -
Then:
python3 apply_review.py review/k60-s24It finds the saved review in Downloads by itself, checks everything again, and names any problem the way the page does ("Group 4: kept, but its name is not finished"). Nothing is written until the review passes; then it writes
situations.json: each situation's sentence and roles, the scenes that show it with their stories, and which groups it came from.
candidates.json in the run directory is the program's record for the checker; leave it alone.
python3 -m unittest test_review # the saved review's checks, the page renderer, apply_review.py
node --test test_review_page.mjs # the rules the page applies as you review
The page's drawing code is only checked for syntax and for starting up; how it looks and behaves in a real browser has to be tried by a person.