JesseMarkowitzandClaude Opus 5
fa3769d0fe
Research toward building a content pack from a story corpus, kept on its own branch and independent of the game. Records the selection experiments against blind labels, and settles selection as gate G2 followed by a human review: review.py writes REVIEW.md and a review.json form, apply_review.py checks the filled form and writes situations.json for the next stage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C6UDQ9o6L6Ey173U7XVou6
91 lines
8.4 KiB
Plaintext
91 lines
8.4 KiB
Plaintext
qwen2.5:3b-instruct: 250 labelled cluster rows; parse failures 1
|
|
theta chosen on dev: 0.6
|
|
dev FIT >= 0.3 accepted 22 >=1 15 label2 4 P1+ 0.68 recall 0.60
|
|
dev FIT >= 0.4 accepted 20 >=1 13 label2 3 P1+ 0.65 recall 0.52
|
|
dev FIT >= 0.5 accepted 16 >=1 11 label2 2 P1+ 0.69 recall 0.44
|
|
dev FIT >= 0.6 accepted 9 >=1 8 label2 2 P1+ 0.89 recall 0.32
|
|
dev FIT >= 0.7 accepted 6 >=1 5 label2 2 P1+ 0.83 recall 0.20
|
|
dev FIT >= 0.8 accepted 4 >=1 4 label2 2 P1+ 1.00 recall 0.16
|
|
dev FIT >= 0.9 accepted 2 >=1 2 label2 1 P1+ 1.00 recall 0.08
|
|
dev FIT >= 1.0 accepted 1 >=1 1 label2 0 P1+ 1.00 recall 0.04
|
|
|
|
mean FIT by label (all rows):
|
|
label 0: n=121 mean 0.39 median 0.40
|
|
label 1: n=106 mean 0.49 median 0.50
|
|
label 2: n= 23 mean 0.42 median 0.33
|
|
|
|
PRIMARY: validate + validate3: 148 clusters, 67 labelled >=1
|
|
all accepted 148 >=1 67 label2 10 P1+ 0.45 recall 1.00
|
|
G2 accepted 49 >=1 36 label2 5 P1+ 0.73 recall 0.54
|
|
G4 FIT >= 0.6 accepted 50 >=1 24 label2 4 P1+ 0.48 recall 0.36
|
|
G4 AND G2 accepted 18 >=1 11 label2 1 P1+ 0.61 recall 0.16
|
|
G4 size <=9 accepted 13 >=1 3 label2 1 P1+ 0.23 recall 0.33 (of 48)
|
|
G4 size 10-25 accepted 24 >=1 12 label2 3 P1+ 0.50 recall 0.39 (of 63)
|
|
G4 size >=26 accepted 13 >=1 9 label2 0 P1+ 0.69 recall 0.33 (of 37)
|
|
|
|
secondary: test + recover: 62 clusters, 37 labelled >=1
|
|
all accepted 62 >=1 37 label2 8 P1+ 0.60 recall 1.00
|
|
G2 accepted 28 >=1 23 label2 5 P1+ 0.82 recall 0.62
|
|
G4 FIT >= 0.6 accepted 21 >=1 15 label2 0 P1+ 0.71 recall 0.41
|
|
G4 AND G2 accepted 14 >=1 12 label2 0 P1+ 0.86 recall 0.32
|
|
G4 size <=9 accepted 1 >=1 0 label2 0 P1+ 0.00 recall 0.00 (of 8)
|
|
G4 size 10-25 accepted 8 >=1 5 label2 0 P1+ 0.62 recall 0.25 (of 30)
|
|
G4 size >=26 accepted 12 >=1 10 label2 0 P1+ 0.83 recall 0.59 (of 24)
|
|
|
|
criterion 1 (P1+ >= 0.80): fail (0.48)
|
|
criterion 2 (more >=1 than G2): fail (24 vs 36)
|
|
criterion 3 (<=9 members P1+ >= 0.67 if >= 3 accepted): fail (13 accepted, P1+ 0.23)
|
|
OVERALL: fail
|
|
|
|
accepted on the primary set, model situation vs label name:
|
|
X003 size 39 label 0 fit 1.00 model: demand and confrontation | label:
|
|
X019 size 45 label 1 fit 1.00 model: tension between unequal parties | label: a public quarrel with a shopkeeper, driver or official
|
|
X067 size 23 label 0 fit 1.00 model: power struggle/conflict over resources | label:
|
|
Y006 size 45 label 1 fit 1.00 model: manipulation and exploitation | label: schemes, cons and theft over money
|
|
Y008 size 5 label 0 fit 1.00 model: power struggle/conflict | label:
|
|
Y017 size 45 label 1 fit 1.00 model: coercion and manipulation for personal gain | label: an invitation to dine
|
|
Y032 size 13 label 1 fit 1.00 model: confrontation over a personal issue | label: a run-in with the police
|
|
Y040 size 6 label 1 fit 1.00 model: people seek social status or attention | label: who holds sway in a dining room or hotel
|
|
Y044 size 5 label 0 fit 1.00 model: dinner party | label:
|
|
Y047 size 18 label 2 fit 1.00 model: competing for status or position | label: working girls: jobs, wages and standing
|
|
Y053 size 45 label 1 fit 1.00 model: confrontation | label: a brawl or showdown
|
|
X035 size 45 label 0 fit 0.93 model: Tension between parties | label:
|
|
X043 size 38 label 1 fit 0.93 model: dinner party/hosting | label: who holds sway in a dining room or hotel
|
|
Y071 size 21 label 1 fit 0.93 model: Negotiation and deal-making | label: schemes, cons and theft over money
|
|
Y074 size 11 label 1 fit 0.91 model: courtship and social status | label: working girls: jobs, wages and standing
|
|
Y019 size 10 label 0 fit 0.90 model: seeking novelty or adventure | label:
|
|
Y063 size 8 label 0 fit 0.88 model: power dynamics in social hierarchies | label:
|
|
Y052 size 23 label 2 fit 0.87 model: dinner plans and dynamics | label: an invitation to dine
|
|
X028 size 6 label 0 fit 0.83 model: power dynamics in social hierarchies | label:
|
|
X012 size 10 label 0 fit 0.80 model: people seeking or offering help or support | label:
|
|
Y001 size 23 label 0 fit 0.80 model: Money disputes and transactions | label:
|
|
Y066 size 17 label 0 fit 0.80 model: power dynamics in luxury vs. poverty | label:
|
|
Y070 size 5 label 0 fit 0.80 model: people pursuing their livelihood or interests | label:
|
|
Y002 size 9 label 0 fit 0.78 model: manipulation and hidden agendas | label:
|
|
X011 size 43 label 1 fit 0.73 model: NEGOTIATION OVER RESOURCES OR SERVICES | label: drinking in the saloon
|
|
X020 size 24 label 0 fit 0.73 model: people hiding or avoiding attention | label:
|
|
X033 size 45 label 0 fit 0.73 model: negotiation and payment | label:
|
|
X036 size 47 label 1 fit 0.73 model: reveal death to family | label: illness, loss and a missing relative
|
|
X048 size 20 label 1 fit 0.73 model: financial transaction or dispute | label: money owed and repayment
|
|
X049 size 25 label 0 fit 0.73 model: coercion or manipulation | label:
|
|
X055 size 21 label 2 fit 0.73 model: marital conflict | label: a spouse schemes against or confronts a spouse over infidelity
|
|
X058 size 16 label 0 fit 0.73 model: welcoming and revealing history | label:
|
|
Y011 size 15 label 0 fit 0.73 model: conflict between individuals seeking different outcomes | label:
|
|
Y022 size 24 label 1 fit 0.73 model: con or scam | label: schemes, cons and theft over money
|
|
Y062 size 45 label 1 fit 0.73 model: Confrontation over a personal relationship | label: a spouse schemes against or confronts a spouse over infidelity
|
|
X039 size 11 label 1 fit 0.73 model: Workplace conflict and exploitation | label: working girls: jobs, wages and standing
|
|
X024 size 7 label 0 fit 0.71 model: NEGOCIACTION | label:
|
|
Y030 size 7 label 1 fit 0.71 model: dining and socializing | label: who holds sway in a dining room or hotel
|
|
X053 size 16 label 0 fit 0.67 model: exploit and exploitation | label:
|
|
X057 size 45 label 1 fit 0.67 model: NEGOTIATION AND POWER PLAY | label: schemes, cons and theft over money
|
|
X059 size 19 label 1 fit 0.67 model: Wedding elopement and deception | label: a proposal or engagement, accepted or refused
|
|
X063 size 6 label 0 fit 0.67 model: confrontation | label:
|
|
Y048 size 12 label 0 fit 0.67 model: seeking advice or guidance | label:
|
|
Y055 size 6 label 0 fit 0.67 model: confrontation | label:
|
|
Y020 size 11 label 0 fit 0.64 model: seeking connections or companionship | label:
|
|
Y029 size 8 label 0 fit 0.62 model: dinner party with hidden agendas | label:
|
|
X041 size 17 label 1 fit 0.60 model: manipulation and control | label: money and marriage: wealthy suitors, inheritances
|
|
X042 size 5 label 2 fit 0.60 model: argue over payment | label: a public quarrel with a shopkeeper, driver or official
|
|
X045 size 25 label 1 fit 0.60 model: seeking/stuck in job/searching employment | label: struggling artists and money
|
|
X073 size 45 label 0 fit 0.60 model: coercion and manipulation | label:
|