JesseMarkowitzandClaude Opus 5.5
3a4758172d
Fourteen corpora split and classified: aesop, bierce, chekhov, holmes, keefe, lawson, lorimer, maupassant, nobody, plaintales, poe, torchy, wallingford and winesburg, each with its splitter and the hand-written groups and pages; catalogue v2 and v3; and the shared splitters gutenberg_chunks.py, se_split.py and se_build.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019rwKTmug58sEsJ72AuWsEi
37 lines
2.6 KiB
Python
37 lines
2.6 KiB
Python
"""Split the Project Gutenberg text of *Three Hundred Aesop's Fables*, translated by George Fyler Townsend (1867,
|
||
eBook #21), one story per fable and ONE SCENE PER FABLE. This departs from the usual scene rule on purpose:
|
||
most fables are under 80 words, so the rule's 80-word tail would discard them, and a fable split in two would
|
||
lose its turn. Titles come from the contents (after its AESOP'S FABLES line); each must be found, in order, as
|
||
a line of its own in the body. The preface and the life of Aesop are dropped. The printed drop capital was lost
|
||
in transcription, so many fables open with the first word's initial letter missing (e.g. "WOLF" for "A WOLF")."""
|
||
import json, pathlib, re, sys
|
||
sys.path.insert(0, '..')
|
||
from gutenberg_chunks import gutenberg_body, paragraphs
|
||
|
||
VOLUME = "Three Hundred Aesop's Fables (tr. George Fyler Townsend, 1867)"
|
||
lines = gutenberg_body('corpus/three-hundred-aesops-fables.txt')
|
||
c0 = next(i for i, l in enumerate(lines) if l.strip() == 'CONTENTS')
|
||
f0 = next(i for i, l in enumerate(lines) if i > c0 and l.strip().upper().replace('Æ', 'AE') in ("AESOP’S FABLES", "AESOP'S FABLES"))
|
||
titles = []
|
||
for l in lines[f0 + 1:]:
|
||
if l.strip(): titles.append(l.strip())
|
||
elif titles and not any(x.strip() for x in lines[lines.index(l) + 1:lines.index(l) + 3]): break
|
||
titles = [t for t in titles if t not in ('FOOTNOTES', 'INDEX')] # back matter listed in the contents
|
||
body0 = next(i for i, l in enumerate(lines) if i > f0 + len(titles) and l.strip() == titles[0])
|
||
heads, at = [], body0
|
||
for t in titles:
|
||
at = next(i for i in range(at, len(lines)) if lines[i].strip() == t and not lines[i].startswith(' '))
|
||
heads.append(at); at += 1
|
||
end = next(i for i in range(heads[-1] + 1, len(lines)) if lines[i].strip() in ('FOOTNOTES', 'INDEX'))
|
||
stories, chunks = [], []
|
||
for n, (t, a) in enumerate(zip(titles, heads)):
|
||
text = ' '.join(paragraphs('\n'.join(lines[a + 1:(heads[n + 1] if n + 1 < len(heads) else end)])))
|
||
w = len(text.split())
|
||
stories.append({'volume': VOLUME, 'title': t, 'words': w, 'text': text})
|
||
chunks.append({'story': n, 'title': t, 'volume': VOLUME, 'text': text, 'words': w})
|
||
pathlib.Path('stories.json').write_text(json.dumps(stories, indent=1, ensure_ascii=False), encoding='utf-8')
|
||
pathlib.Path('chunks.json').write_text(json.dumps(chunks, ensure_ascii=False), encoding='utf-8')
|
||
ws = sorted(c['words'] for c in chunks)
|
||
print(f'fables {len(stories)} words/fable min {ws[0]} median {ws[len(ws)//2]} max {ws[-1]} total {sum(ws):,}')
|
||
print('shortest:', min(stories, key=lambda s: s['words'])['title'], '| longest:', max(stories, key=lambda s: s['words'])['title'])
|