JesseMarkowitzandClaude Opus 5.5
3a4758172d
Fourteen corpora split and classified: aesop, bierce, chekhov, holmes, keefe, lawson, lorimer, maupassant, nobody, plaintales, poe, torchy, wallingford and winesburg, each with its splitter and the hand-written groups and pages; catalogue v2 and v3; and the shared splitters gutenberg_chunks.py, se_split.py and se_build.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019rwKTmug58sEsJ72AuWsEi
46 lines
2.8 KiB
Python
46 lines
2.8 KiB
Python
"""Split three Project Gutenberg volumes of Sewell Ford's Torchy stories into one corpus, one story per
|
|
chapter (each chapter is a self-contained episode): *Torchy* (#20626), *On With Torchy* (#17301) and
|
|
*Torchy, Private Sec.* (#20627), in that order, which is the order of his rise from office boy.
|
|
Chapter titles come from each volume's contents; the body heads each chapter with a bare
|
|
`CHAPTER <roman>` line, and those must number 1..N with N equal to the contents count.
|
|
|
|
KNOWN DEFECT, kept on purpose: volumes 1 and 3 end in the publisher's advertisements, which this script
|
|
does not cut, so they ran into the last chapter of each. Chunks 245, 736 and 737 are advertising only;
|
|
chunks 244 and 735 are the real end of a story followed by the start of the advertising. They are listed in
|
|
NOT_STORY.json and excluded from grouping. The script was not changed afterwards, because readers had
|
|
already written records against these chunk numbers and cutting the advertising in volume 1 would shift
|
|
every later id. Fix it (stop each volume at its GROSSET & DUNLAP line) only together with a renumbering
|
|
of claude/batches."""
|
|
import re, sys
|
|
sys.path.insert(0, '..')
|
|
from gutenberg_chunks import gutenberg_body, paragraphs, write
|
|
|
|
VOLUMES = [('corpus/torchy.txt', 'Torchy (Sewell Ford, 1911)'),
|
|
('corpus/on-with-torchy.txt', 'On With Torchy (Sewell Ford, 1914)'),
|
|
('corpus/torchy-private-sec.txt', 'Torchy, Private Sec. (Sewell Ford, 1917)')]
|
|
ROMAN = {'I': 1, 'V': 5, 'X': 10, 'L': 50}
|
|
def roman(s): return sum(-ROMAN[a] if ROMAN[a] < ROMAN.get(b, 0) else ROMAN[a] for a, b in zip(s, s[1:] + ' '))
|
|
|
|
stories = []
|
|
for path, volume in VOLUMES:
|
|
lines = gutenberg_body(path)
|
|
c0 = next(i for i, l in enumerate(lines) if l.strip() == 'CONTENTS')
|
|
contents = []
|
|
for l in lines[c0 + 1:]:
|
|
m = re.fullmatch(r'\s*([IVXL]+)\.\s+(.+?)(\s+\d+)?\s*', l)
|
|
if m: contents.append(m.group(2).strip())
|
|
elif contents and not l.strip(): continue
|
|
elif contents and l.strip(): break
|
|
heads = [i for i, l in enumerate(lines) if re.fullmatch(r'CHAPTER [IVXL]+', l.strip())]
|
|
nums = [roman(lines[i].split()[1]) for i in heads]
|
|
assert nums == list(range(1, len(contents) + 1)), (path, nums, len(contents))
|
|
ends = heads[1:] + [len(lines)]
|
|
for n, (a, b) in enumerate(zip(heads, ends)):
|
|
body = lines[a + 1:b]
|
|
k = next(j for j, l in enumerate(body) if l.strip()) # the chapter title line(s) under the heading
|
|
while k < len(body) and body[k].strip() and body[k].strip().upper() == body[k].strip(): k += 1
|
|
paras = paragraphs('\n'.join(body[k:]), drop=(r'THE END\.?', r'\[Illustration[^\]]*\]'))
|
|
stories.append({'volume': volume, 'title': contents[n].title() if contents[n].isupper() else contents[n],
|
|
'words': sum(len(p.split()) for p in paras), 'text': '\n\n'.join(paras)})
|
|
write('.', stories)
|