story-to-pack: add the corpora and the catalogue v2/v3

Fourteen corpora split and classified: aesop, bierce, chekhov, holmes,
keefe, lawson, lorimer, maupassant, nobody, plaintales, poe, torchy,
wallingford and winesburg, each with its splitter and the hand-written
groups and pages; catalogue v2 and v3; and the shared splitters
gutenberg_chunks.py, se_split.py and se_build.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019rwKTmug58sEsJ72AuWsEi
This commit is contained in:
JesseMarkowitz
2026-10-07 06:05:51 -04:00
co-authored by Claude Opus 5.5
parent e5617b86ba
commit 3a4758172d
566 changed files with 303901 additions and 0 deletions
@@ -0,0 +1,36 @@
"""Split the Project Gutenberg text of *Three Hundred Aesop's Fables*, translated by George Fyler Townsend (1867,
eBook #21), one story per fable and ONE SCENE PER FABLE. This departs from the usual scene rule on purpose:
most fables are under 80 words, so the rule's 80-word tail would discard them, and a fable split in two would
lose its turn. Titles come from the contents (after its AESOP'S FABLES line); each must be found, in order, as
a line of its own in the body. The preface and the life of Aesop are dropped. The printed drop capital was lost
in transcription, so many fables open with the first word's initial letter missing (e.g. "WOLF" for "A WOLF")."""
import json, pathlib, re, sys
sys.path.insert(0, '..')
from gutenberg_chunks import gutenberg_body, paragraphs
VOLUME = "Three Hundred Aesop's Fables (tr. George Fyler Townsend, 1867)"
lines = gutenberg_body('corpus/three-hundred-aesops-fables.txt')
c0 = next(i for i, l in enumerate(lines) if l.strip() == 'CONTENTS')
f0 = next(i for i, l in enumerate(lines) if i > c0 and l.strip().upper().replace('Æ', 'AE') in ("AESOP’S FABLES", "AESOP'S FABLES"))
titles = []
for l in lines[f0 + 1:]:
if l.strip(): titles.append(l.strip())
elif titles and not any(x.strip() for x in lines[lines.index(l) + 1:lines.index(l) + 3]): break
titles = [t for t in titles if t not in ('FOOTNOTES', 'INDEX')] # back matter listed in the contents
body0 = next(i for i, l in enumerate(lines) if i > f0 + len(titles) and l.strip() == titles[0])
heads, at = [], body0
for t in titles:
at = next(i for i in range(at, len(lines)) if lines[i].strip() == t and not lines[i].startswith(' '))
heads.append(at); at += 1
end = next(i for i in range(heads[-1] + 1, len(lines)) if lines[i].strip() in ('FOOTNOTES', 'INDEX'))
stories, chunks = [], []
for n, (t, a) in enumerate(zip(titles, heads)):
text = ' '.join(paragraphs('\n'.join(lines[a + 1:(heads[n + 1] if n + 1 < len(heads) else end)])))
w = len(text.split())
stories.append({'volume': VOLUME, 'title': t, 'words': w, 'text': text})
chunks.append({'story': n, 'title': t, 'volume': VOLUME, 'text': text, 'words': w})
pathlib.Path('stories.json').write_text(json.dumps(stories, indent=1, ensure_ascii=False), encoding='utf-8')
pathlib.Path('chunks.json').write_text(json.dumps(chunks, ensure_ascii=False), encoding='utf-8')
ws = sorted(c['words'] for c in chunks)
print(f'fables {len(stories)} words/fable min {ws[0]} median {ws[len(ws)//2]} max {ws[-1]} total {sum(ws):,}')
print('shortest:', min(stories, key=lambda s: s['words'])['title'], '| longest:', max(stories, key=lambda s: s['words'])['title'])