Jesse.Markowitz/TheLadder
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add story-to-pack research and a structured situation review

Browse Source 
Research toward building a content pack from a story corpus, kept on its own
branch and independent of the game. Records the selection experiments against
blind labels, and settles selection as gate G2 followed by a human review:
review.py writes REVIEW.md and a review.json form, apply_review.py checks the
filled form and writes situations.json for the next stage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C6UDQ9o6L6Ey173U7XVou6
This commit is contained in:
JesseMarkowitz
2026-09-15 06:59:35 -04:00
co-authored by Claude Opus 5
parent d84ae495f4
commit fa3769d0fe
112 changed files with 79868 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/story-to-pack/probe/evaluate-judge-14b.log
+82
View File
@@ -0,0 +1,82 @@
qwen3:14b: 250 labelled cluster rows; parse failures 0
theta chosen on dev: 0.6
dev FIT >= 0.3 accepted 29 >=1 19 label2 5 P1+ 0.66 recall 0.76
dev FIT >= 0.4 accepted 28 >=1 18 label2 5 P1+ 0.64 recall 0.72
dev FIT >= 0.5 accepted 14 >=1 10 label2 2 P1+ 0.71 recall 0.40
dev FIT >= 0.6 accepted 8 >=1 7 label2 2 P1+ 0.88 recall 0.28
dev FIT >= 0.7 accepted 4 >=1 4 label2 1 P1+ 1.00 recall 0.16
dev FIT >= 0.8 accepted 2 >=1 2 label2 0 P1+ 1.00 recall 0.08
dev FIT >= 0.9 accepted 0 >=1 0 label2 0 P1+ 0.00 recall 0.00
dev FIT >= 1.0 accepted 0 >=1 0 label2 0 P1+ 0.00 recall 0.00
mean FIT by label (all rows):
label 0: n=121 mean 0.40 median 0.42
label 1: n=106 mean 0.47 median 0.47
label 2: n= 23 mean 0.52 median 0.53
PRIMARY: validate + validate3: 148 clusters, 67 labelled >=1
all accepted 148 >=1 67 label2 10 P1+ 0.45 recall 1.00
G2 accepted 49 >=1 36 label2 5 P1+ 0.73 recall 0.54
G4 FIT >= 0.6 accepted 42 >=1 22 label2 4 P1+ 0.52 recall 0.33
G4 AND G2 accepted 10 >=1 10 label2 1 P1+ 1.00 recall 0.15
G4 size <=9 accepted 17 >=1 5 label2 2 P1+ 0.29 recall 0.56 (of 48)
G4 size 10-25 accepted 14 >=1 7 label2 2 P1+ 0.50 recall 0.23 (of 63)
G4 size >=26 accepted 11 >=1 10 label2 0 P1+ 0.91 recall 0.37 (of 37)
secondary: test + recover: 62 clusters, 37 labelled >=1
all accepted 62 >=1 37 label2 8 P1+ 0.60 recall 1.00
G2 accepted 28 >=1 23 label2 5 P1+ 0.82 recall 0.62
G4 FIT >= 0.6 accepted 15 >=1 9 label2 1 P1+ 0.60 recall 0.24
G4 AND G2 accepted 11 >=1 8 label2 1 P1+ 0.73 recall 0.22
G4 size <=9 accepted 1 >=1 0 label2 0 P1+ 0.00 recall 0.00 (of 8)
G4 size 10-25 accepted 5 >=1 3 label2 1 P1+ 0.60 recall 0.15 (of 30)
G4 size >=26 accepted 9 >=1 6 label2 0 P1+ 0.67 recall 0.35 (of 24)
criterion 1 (P1+ >= 0.80): fail (0.52)
criterion 2 (more >=1 than G2): fail (22 vs 36)
criterion 3 (<=9 members P1+ >= 0.67 if >= 3 accepted): fail (17 accepted, P1+ 0.29)
OVERALL: fail
accepted on the primary set, model situation vs label name:
Y026 size 6 label 0 fit 1.00 model: Someone wants something from someone else, but there is | label:
Y052 size 23 label 2 fit 1.00 model: Planning or attending a dinner with hidden intentions | label: an invitation to dine
X002 size 10 label 1 fit 0.90 model: Someone seeks artistic validation or support from anoth | label: struggling artists and money
Y010 size 18 label 1 fit 0.87 model: Artist seeking financial support or patronage | label: struggling artists and money
Y053 size 45 label 1 fit 0.87 model: Conflict between individuals over personal issues | label: a brawl or showdown
X015 size 38 label 0 fit 0.80 model: Seeking something from someone in a city setting | label:
X030 size 18 label 1 fit 0.80 model: Confrontation between individuals or groups leading to | label: a brawl or showdown
X055 size 21 label 2 fit 0.80 model: Marital conflict and tension | label: a spouse schemes against or confronts a spouse over infidelity
Y008 size 5 label 0 fit 0.80 model: Someone holds power over someone else | label:
Y011 size 15 label 0 fit 0.80 model: Someone confronts someone else over a personal issue | label:
Y034 size 5 label 0 fit 0.80 model: conflict between perception and reality | label:
X038 size 8 label 0 fit 0.75 model: Struggling to find acceptance or connection in a hostil | label:
X048 size 20 label 1 fit 0.73 model: Financial transaction with potential conflict | label: money owed and repayment
X058 size 16 label 0 fit 0.73 model: A host welcomes a visitor with mixed emotions and inten | label:
Y017 size 45 label 1 fit 0.73 model: Invitation to dinner with hidden intentions | label: an invitation to dine
Y073 size 36 label 1 fit 0.73 model: Someone demands information or action from another, who | label: writers and editors
X013 size 7 label 1 fit 0.71 model: Seeking financial advice | label: what to do with money, and whose advice to take
X024 size 7 label 0 fit 0.71 model: Demanding repayment or compensation | label:
Y030 size 7 label 1 fit 0.71 model: Social interaction at a dining establishment | label: who holds sway in a dining room or hotel
X020 size 24 label 0 fit 0.67 model: secretive interactions between individuals | label:
X050 size 45 label 1 fit 0.67 model: Financial transactions involving coercion or manipulati | label: schemes, cons and theft over money
X059 size 19 label 1 fit 0.67 model: Marriage proposals and preparations | label: a proposal or engagement, accepted or refused
X068 size 6 label 2 fit 0.67 model: Marital conflict and confrontation | label: a spouse schemes against or confronts a spouse over infidelity
X072 size 17 label 0 fit 0.67 model: Resentment over unreciprocated affection | label:
Y031 size 6 label 0 fit 0.67 model: Older person mentoring younger person | label:
Y036 size 6 label 0 fit 0.67 model: envy and desire for wealth or power | label:
Y040 size 6 label 1 fit 0.67 model: Competing for social status through dining experiences | label: who holds sway in a dining room or hotel
Y068 size 6 label 0 fit 0.67 model: Conflict between individuals with opposing goals | label:
X010 size 11 label 0 fit 0.64 model: competition for status or position in a city | label:
Y050 size 8 label 0 fit 0.62 model: Someone trying to win someone else's affection | label:
Y063 size 8 label 0 fit 0.62 model: Power dynamics between individuals | label:
X009 size 45 label 1 fit 0.60 model: conflict over marriage proposals and social expectation | label: a proposal or engagement, accepted or refused
X012 size 10 label 0 fit 0.60 model: A person of high status seeking to help someone in need | label:
X014 size 10 label 0 fit 0.60 model: Someone threatens someone else | label:
X037 size 5 label 0 fit 0.60 model: conflict between individuals due to past relationships | label:
X042 size 5 label 2 fit 0.60 model: Dispute over payment for a service | label: a public quarrel with a shopkeeper, driver or official
X043 size 38 label 1 fit 0.60 model: Dinner party with social dynamics | label: who holds sway in a dining room or hotel
X056 size 45 label 1 fit 0.60 model: Attempting to gain favor or influence through social in | label: trying to impress or win someone's favour
X057 size 45 label 1 fit 0.60 model: Someone tries to manipulate someone else for personal g | label: schemes, cons and theft over money
Y021 size 45 label 1 fit 0.60 model: Someone manipulates someone into a transaction for pers | label: schemes, cons and theft over money
Y033 size 5 label 0 fit 0.60 model: Someone is suspected of a crime they did not commit | label:
Y043 size 31 label 1 fit 0.60 model: Confrontation over a dispute or accusation | label: a run-in with the police