story-to-pack: add the corpora and the catalogue v2/v3
Fourteen corpora split and classified: aesop, bierce, chekhov, holmes, keefe, lawson, lorimer, maupassant, nobody, plaintales, poe, torchy, wallingford and winesburg, each with its splitter and the hand-written groups and pages; catalogue v2 and v3; and the shared splitters gutenberg_chunks.py, se_split.py and se_build.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019rwKTmug58sEsJ72AuWsEi
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
e5617b86ba
commit
3a4758172d
@@ -0,0 +1,36 @@
|
||||
"""Split the Project Gutenberg text of *Three Hundred Aesop's Fables*, translated by George Fyler Townsend (1867,
|
||||
eBook #21), one story per fable and ONE SCENE PER FABLE. This departs from the usual scene rule on purpose:
|
||||
most fables are under 80 words, so the rule's 80-word tail would discard them, and a fable split in two would
|
||||
lose its turn. Titles come from the contents (after its AESOP'S FABLES line); each must be found, in order, as
|
||||
a line of its own in the body. The preface and the life of Aesop are dropped. The printed drop capital was lost
|
||||
in transcription, so many fables open with the first word's initial letter missing (e.g. "WOLF" for "A WOLF")."""
|
||||
import json, pathlib, re, sys
|
||||
sys.path.insert(0, '..')
|
||||
from gutenberg_chunks import gutenberg_body, paragraphs
|
||||
|
||||
VOLUME = "Three Hundred Aesop's Fables (tr. George Fyler Townsend, 1867)"
|
||||
lines = gutenberg_body('corpus/three-hundred-aesops-fables.txt')
|
||||
c0 = next(i for i, l in enumerate(lines) if l.strip() == 'CONTENTS')
|
||||
f0 = next(i for i, l in enumerate(lines) if i > c0 and l.strip().upper().replace('Æ', 'AE') in ("AESOP’S FABLES", "AESOP'S FABLES"))
|
||||
titles = []
|
||||
for l in lines[f0 + 1:]:
|
||||
if l.strip(): titles.append(l.strip())
|
||||
elif titles and not any(x.strip() for x in lines[lines.index(l) + 1:lines.index(l) + 3]): break
|
||||
titles = [t for t in titles if t not in ('FOOTNOTES', 'INDEX')] # back matter listed in the contents
|
||||
body0 = next(i for i, l in enumerate(lines) if i > f0 + len(titles) and l.strip() == titles[0])
|
||||
heads, at = [], body0
|
||||
for t in titles:
|
||||
at = next(i for i in range(at, len(lines)) if lines[i].strip() == t and not lines[i].startswith(' '))
|
||||
heads.append(at); at += 1
|
||||
end = next(i for i in range(heads[-1] + 1, len(lines)) if lines[i].strip() in ('FOOTNOTES', 'INDEX'))
|
||||
stories, chunks = [], []
|
||||
for n, (t, a) in enumerate(zip(titles, heads)):
|
||||
text = ' '.join(paragraphs('\n'.join(lines[a + 1:(heads[n + 1] if n + 1 < len(heads) else end)])))
|
||||
w = len(text.split())
|
||||
stories.append({'volume': VOLUME, 'title': t, 'words': w, 'text': text})
|
||||
chunks.append({'story': n, 'title': t, 'volume': VOLUME, 'text': text, 'words': w})
|
||||
pathlib.Path('stories.json').write_text(json.dumps(stories, indent=1, ensure_ascii=False), encoding='utf-8')
|
||||
pathlib.Path('chunks.json').write_text(json.dumps(chunks, ensure_ascii=False), encoding='utf-8')
|
||||
ws = sorted(c['words'] for c in chunks)
|
||||
print(f'fables {len(stories)} words/fable min {ws[0]} median {ws[len(ws)//2]} max {ws[-1]} total {sum(ws):,}')
|
||||
print('shortest:', min(stories, key=lambda s: s['words'])['title'], '| longest:', max(stories, key=lambda s: s['words'])['title'])
|
||||
Reference in New Issue
Block a user