Jesse.Markowitz/interactive-story
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite Python comments in Google developer documentation style (#12)

Browse Source 
* Rewrite comments in Google developer documentation style

Rewrite the comments and docstrings across the backend core modules so they
read plainly. The previous prose was accurate but dense and figurative, which
made it slow to skim.

Applies the Google developer documentation style guide: short sentences, active
voice, present tense, American spelling, and no metaphors, idioms, or
rhetorical asides. Replaces em-dash chains with separate sentences.
This commit is contained in:
Parth
2026-08-26 15:37:25 +05:30
committed by GitHub
parent cf6161a5ee
commit e7d75c3b05
83 changed files with 4605 additions and 3988 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/context/builder.py
+112 -98
View File
@@ -10,11 +10,11 @@
[Author's Note] injected AUTHORS_NOTE_DEPTH actions before the end of history
[Latest player action] (+ script frontMemory right after it, Phase 4)
Which components are present is AI Dungeon's design, above. The *order* they
are laid out in is not: everything fixed is emitted first and everything that
moves after the history, because prompt caching bills on a shared prefix and
one mutable section high up re-prices the whole prompt under it. See the two
"static block" / "live sections" comments in `build_context`.
The list above comes from AI Dungeon's design. The order does not. This module
emits every fixed section first and every changing section after the history,
because prompt caching bills on a shared prefix. A section that changes near the
top of the prompt re-prices everything below it. See the comments on the static
block and the live sections in `build_context`.
"""
import functools
@@ -30,33 +30,33 @@ CARD_BUDGET_SHARE = 0.4 # max share of non-reserved budget that story cards may
NPC_WINDOW = 6 # actions of story searched for NPC trigger words ("in scene")
SEPARATOR = "\n\n"
# Output-length guidance. max_output_tokens is a hard wall the endpoint enforces
# mid-sentence: hitting it truncates whatever is being written, and since the
# state block is emitted last, it is what gets lost. Asking the model to land
# just inside the wall keeps the cut from happening in the first place.
LENGTH_HEADROOM = 50 # tokens held back from the cap for the state block itself
# Output-length guidance. The endpoint enforces `max_output_tokens` as a hard
# limit, and it truncates the reply mid-sentence when the model reaches it. The
# state block is emitted last, so truncation removes it. Asking the model to
# finish inside the limit prevents the truncation.
LENGTH_HEADROOM = 50 # Tokens reserved from the cap for the state block.
# Models cannot count their own tokens, but they do follow a word budget, so the
# reserved budget is stated in words. ~0.75 words per token for English prose.
# hint states a number of words. English prose averages 0.75 words per token.
WORDS_PER_TOKEN = 0.75
# A word budget is a suggestion the model routinely overshoots, and the cap it is
# protecting is a hard wall — so aim 10% short of the real ceiling and let the
# overshoot land in the slack instead of in the state block.
# Models regularly exceed a word budget, and the cap it protects is a hard
# limit. Aiming 10% below the real ceiling leaves room for that overshoot, so it
# does not consume the state block.
LENGTH_BUFFER = 0.90
MIN_LENGTH_HINT_WORDS = 40 # below this the hint is noise; a tiny cap speaks for itself
# A ceiling alone is a one-sided instruction, and models read it very differently:
# a verbose one is held back by it, while a terse one has nothing to act on except
# the "write only as much as the moment needs" clause and collapses to two
# paragraphs. Stating a floor as well turns the guidance into a band, so the same
# prompt lands in the same place regardless of which way the model leans. Set as a
# share of the ceiling so the floor can never approach it.
MIN_LENGTH_HINT_WORDS = 40 # Below this, the hint adds nothing useful.
# A ceiling on its own gives one-sided guidance, and models respond to it
# differently. A verbose model treats it as a limit. A terse model has only the
# instruction to write as much as the moment needs, and it produces two
# paragraphs. Adding a floor turns the guidance into a range, so the same prompt
# produces a similar length from either model. The floor is a share of the
# ceiling so that it can never approach the ceiling.
LENGTH_FLOOR_SHARE = 0.35
# Below this a floor is meaningless — at a tight cap a short turn is the correct
# turn — and the tight-cap wording is the one measured to keep the state block
# alive, so it is left exactly as it was.
# Below this word count, a floor means nothing, because a short turn is the
# correct turn at a tight cap. The wording used at a tight cap is also the
# wording that was measured to preserve the state block, so it is unchanged.
MIN_LENGTH_FLOOR_WORDS = 60
# The floor exists to stop a collapse to two paragraphs, not to demand an essay:
# at a 2400-token cap the share alone would ask for 555 words *minimum*. Past this
# point a reader wanting more length can say so in the author's note.
# The floor prevents a collapse to two paragraphs. It does not ask for an essay.
# At a 2400-token cap, the share alone would request a minimum of 555 words. A
# reader who wants longer turns can ask for them in the author's note.
MAX_LENGTH_FLOOR_WORDS = 300
@@ -89,9 +89,9 @@ class Section:
def length_hint(max_output_tokens: int, *, has_ws: bool) -> str:
"""Ask for a turn that fits inside the output cap, stated as a word budget.
Returns "" when the cap is too small to phrase usefully — the hint is a
suggestion the model can drift past, so it only earns its tokens when there
is enough room for the drift to still land inside the wall.
Returns an empty string when the cap is too small to state usefully. The
model can exceed the hint, so the hint earns its tokens only when there is
enough room for that overshoot to stay inside the cap.
"""
words = int((max_output_tokens - LENGTH_HEADROOM) * WORDS_PER_TOKEN * LENGTH_BUFFER)
if words < MIN_LENGTH_HINT_WORDS:
@@ -102,25 +102,27 @@ def length_hint(max_output_tokens: int, *, has_ws: bool) -> str:
else " Bring the turn to a close well inside the limit rather than "
"stopping mid-sentence."
)
# Phrased as a ceiling, never as a budget. Measured against this model, "keep
# this turn under about N words" reads as a target to fill: it moved a 174-word
# average to 246 (n=5, every run longer than every unhinted one), i.e. the hint
# pushed turns toward the very wall it exists to keep them away from. Naming
# the number as a limit, plus saying a typical turn is far shorter, left the
# average at 170 while still rescuing the state block at tight caps.
# State the number as a ceiling, never as a budget. In measurements, the
# wording "keep this turn under about N words" read to the model as a target
# to fill. It raised the average from 174 words to 246 across five runs, and
# every hinted run was longer than every unhinted run. The hint therefore
# pushed turns toward the limit it exists to avoid. Naming the number as a
# limit, and adding that a typical turn is much shorter, held the average at
# 170 while still preserving the state block at tight caps.
floor = min(int(words * LENGTH_FLOOR_SHARE), MAX_LENGTH_FLOOR_WORDS)
if floor < MIN_LENGTH_FLOOR_WORDS:
return (
f"[Hard limit: this turn must not exceed {words} words. Write only as "
f"much as the moment needs — a typical turn is much shorter.{tail}]"
)
# Both numbers are bounds, and deliberately asymmetric ones: "must not exceed"
# for the wall the endpoint enforces, "should not stop short of" for the floor.
# Neither is a target, which is what the measurement above says matters. The
# "prefer the lower end" clause does the job the old "a typical turn is much
# shorter" line did — holding a verbose model off the wall — but now with a
# number under it, so a terse model reading the same clause lands on the floor
# instead of at forty words.
# Both numbers are bounds, and the wording is deliberately asymmetric. The
# ceiling uses "must not exceed", because the endpoint enforces it. The floor
# uses "should not stop short of". Neither reads as a target, which the
# measurement above shows is what matters. The clause that asks the model to
# prefer the lower end does the job the earlier wording did, which was to
# keep a verbose model away from the ceiling. It now has a number beneath it,
# so a terse model reading the same clause stops at the floor rather than at
# forty words.
return (
f"[Hard limit: this turn must not exceed {words} words, and it should not "
f"stop short of about {floor}. Prefer the lower end of that range unless "
@@ -136,16 +138,19 @@ def _script_memory(adventure: models.Adventure) -> dict:
def _history_text(action: models.Action) -> str:
"""An AI turn as the model should see it in replayed history: its narration
with the state block it emitted re-appended (reconstructed from the stored
delta). The block is stripped before storage/UI, so without this every past
AI turn would look like one that emitted nothing — biasing the model, by
imitation, to stop emitting too. Player turns and blockless turns are
returned unchanged.
"""Returns an AI turn as the model should see it in replayed history.
Reads `world_delta`, not `context_snapshot`: this runs for every action in
the replayed history, and the snapshot is deferred precisely so a turn
never drags the prompt archive out of the database."""
The result is the narration with its state block appended again,
reconstructed from the stored delta. The app strips that block before
storing and displaying the turn. Without this function, every past AI turn
would appear to have emitted no state, and the model would copy that pattern
and stop emitting state itself. Player turns and turns with no block pass
through unchanged.
This function reads `world_delta` rather than `context_snapshot`. It runs
for every action in the replayed history, and `context_snapshot` is deferred
so that a turn never loads the prompt archive from the database.
"""
text = action.text
wd = action.world_delta if isinstance(action.world_delta, dict) else None
if wd:
@@ -156,10 +161,13 @@ def _history_text(action: models.Action) -> str:
def _visible_npcs(actions: list[models.Action], stat_schema: dict) -> dict[str, str]:
"""Defined NPCs whose trigger words appear in the recent story — the ones
"in scene", so only their stats get injected. Maps npc id -> display name.
"""Returns the NPCs whose trigger words appear in the recent story.
`actions` is already the last handful (see NPC_WINDOW)."""
These are the NPCs in scene, and the prompt includes stats for them only.
The result maps an NPC id to its display name.
`actions` holds only the most recent actions. See `NPC_WINDOW`.
"""
recent = SEPARATOR.join(a.text for a in actions).lower()
visible: dict[str, str] = {}
for npc_key, ndef in (stat_schema.get("npcs") or {}).items():
@@ -171,8 +179,11 @@ def _visible_npcs(actions: list[models.Action], stat_schema: dict) -> dict[str,
def _match_cards(cards: list[models.StoryCard], window_text: str) -> list[dict]:
"""AI Dungeon trigger rules: case-insensitive, space-sensitive, partial-word
('boat' triggers on 'boats'). Returns one record per card with the keyword that fired."""
"""Returns one record per matched story card, naming the keyword that matched.
Matching follows AI Dungeon's rules. It ignores case, respects spaces, and
matches partial words, so "boat" matches "boats".
"""
haystack = window_text.lower()
matched = []
for card in cards:
@@ -196,20 +207,19 @@ def build_context(
`exclude_action_id` omits one action from the story (see history.py)."""
script_mem = _script_memory(adventure)
# ----- The static block: byte-for-byte the same prompt every turn -----
# The ordering here is a billing decision, not a stylistic one. Prompt
# caching matches a *prefix*: an endpoint reuses the prompt up to the first
# byte that differs from last time and no further. So one mutable section
# near the top re-prices everything below it, and what is below it is the
# story history, which is the bulk of the prompt. Anything that changes
# turn to turn therefore goes *after* the history, in the live sections —
# which is also where recency serves it best, the same reasoning that
# already puts EMIT_REMINDER last.
# ----- The static block, which is identical on every turn -----
# This ordering exists to reduce cost. Prompt caching matches a prefix. The
# endpoint reuses the prompt up to the first byte that differs from the
# previous request, and no further. A section that changes near the top
# therefore re-prices everything below it, and what sits below it is the
# story history, which is most of the prompt. Sections that change from turn
# to turn go after the history, among the live sections. Placing them there
# also gives them the most recency, which is why `EMIT_REMINDER` goes last.
system_sections: list[Section] = [Section("narrator", settings.narrator_prompt.strip())]
# RPG world state (Phase 12): how to report changes. The live values are a
# live section below; the schema-derived guide and the emit rule are fixed
# for as long as the scenario is.
# RPG world state (Phase 12): the instructions for reporting changes. The
# live values go into a live section below. The guide derived from the
# schema and the emit rule do not change while the scenario is unchanged.
stat_schema = adventure.scenario.stat_schema if adventure.scenario else None
has_ws = worldstate.has_schema(stat_schema)
if has_ws:
@@ -227,13 +237,14 @@ def build_context(
Section("plot_essentials", f"Plot essentials:\n{adventure.memory.strip()}")
)
# ----- Live sections: everything that moves, built here, placed after the
# history further down. Ordered least-volatile first, so a turn that
# changes only the fastest-moving one keeps the others cached too: the
# summary is rewritten every few turns, lore turns over with the scene, the
# retrieved memories change on most turns, the stat values on nearly all.
# `world_lore` joins them below — it is the history window that triggers
# the cards, so it cannot be known yet.
# ----- Live sections, which hold everything that changes -----
# This code builds them here and places them after the history further down.
# They are ordered from least to most volatile, so a turn that changes only
# the fastest-moving section leaves the others cached. The summary is
# rewritten every few turns. Lore changes with the scene. The retrieved
# memories change on most turns, and the stat values change on nearly every
# turn. `world_lore` is added below, because the history window determines
# which cards trigger and that window is not known yet.
summary_section = (
Section("story_summary", f"Story summary:\n{adventure.story_summary.strip()}")
if adventure.story_summary.strip()
@@ -265,9 +276,9 @@ def build_context(
length_note = length_hint(settings.max_output_tokens, has_ws=has_ws)
# The live sections moved below the history but they are still in the
# prompt, so they are still reserved against the budget. (`world_lore` is
# not: it is budgeted out of `available` further down, as it always was.)
# The live sections sit below the history, but they are still part of the
# prompt, so they still count against the budget. `world_lore` is the
# exception, because the code below budgets it out of `available`.
reserved = (
sum(s.tokens for s in system_sections)
+ sum(
@@ -282,10 +293,11 @@ def build_context(
)
available = max(256, settings.context_token_budget - reserved)
# Only the newest actions can reach the prompt: everything below is either
# truncated to `available` tokens or stops at the budget. Fetch a window
# that is provably larger than that and no more — a long adventure would
# otherwise read its entire history every turn to use the tail of it.
# Only the newest actions can reach the prompt, because the code below
# either truncates the text to `available` tokens or stops at the budget.
# Fetch a window that is provably larger than that and no larger. Otherwise
# a long adventure reads its whole history on every turn and uses only the
# end of it.
actions = history.window_covering(
adventure, available, count_tokens, exclude_action_id
)
@@ -319,8 +331,8 @@ def build_context(
spent = 0
oldest_truncated = False
for action in reversed(actions):
# Budget on the text as it will actually appear — with the re-attached
# state block (B) when this adventure tracks world state.
# Budget against the text as it appears in the prompt, which includes
# the state block when this adventure tracks world state.
rendered = _history_text(action) if has_ws else action.text
tokens = count_tokens(rendered) + count_tokens(SEPARATOR)
if spent + tokens > history_budget:
@@ -339,9 +351,9 @@ def build_context(
spent += tokens
included_actions.reverse()
# ----- Assemble story text with author's note near the end -----
# Re-attach each AI turn's state block (stripped before storage) so recent
# history shows the model its own emit pattern to imitate.
# ----- Assemble the story text, with the author's note near the end -----
# Append each AI turn's state block again. The app strips it before storage,
# and the recent history has to show the model the pattern to follow.
texts = [_history_text(a) if has_ws else a.text for a in included_actions]
note_sections: list[Section] = []
if authors_note:
@@ -353,21 +365,22 @@ def build_context(
note_sections.append(Section("recent_history", SEPARATOR.join(after)))
else:
note_sections.append(Section("history", SEPARATOR.join(texts)))
# The live sections, least volatile first (see where they are built). They
# sit below the history so the history caches, and above the tail so the
# three sections that are last for a reason stay last.
# The live sections, ordered from least to most volatile. See the comment
# where they are built. They go below the history so that the history stays
# cached, and above the final sections so that those stay last.
for live in (summary_section, lore_section, memories_section, world_state_section):
if live is not None:
note_sections.append(live)
if front_memory:
note_sections.append(Section("front_memory", front_memory))
# Sits just above the emit reminder, which keeps the strongest recency slot:
# the length budget is about the narration, the reminder about the block that
# comes after it, so this is also the order the model has to act in.
# Place the length hint just above the emit reminder, which keeps the last
# position. The length budget applies to the narration, and the reminder
# applies to the block that follows it, so this is also the order in which
# the model acts.
note_sections.append(Section("length_hint", length_note))
if has_ws:
# Terminal reminder: the emit rule sits up in the system block, far from
# where the model generates; repeat it last, in the strongest recency slot.
# The emit rule sits in the system block, far from where the model
# generates text, so repeat it last where it has the most effect.
note_sections.append(Section("world_state_reminder", worldstate.EMIT_REMINDER))
story_sections = [s for s in note_sections if s.text]
@@ -388,8 +401,9 @@ def build_context(
"memories": memory_bank,
"history": {
"included": len(included_actions),
# The whole story, not just the window fetched above — Insights
# reports "N of M actions included" and M is the real total.
# The count covers the whole story rather than the window fetched
# above. Insights reports how many of the total actions it
# included, so this number must be the real total.
"total": history.count(adventure, exclude_action_id),
"oldest_truncated": oldest_truncated,
},
backend/app/context/cursors.py
+73 -69
View File
@@ -1,31 +1,32 @@
"""Phase 14 — how far along a story the derived work has got.
"""Phase 14: tracks how far along a story the derived work has reached.
Two things are built from the story and stored beside it: the memories, and the
Story Summary. Both need to know where they left off, and that mark used to be
a *count* — "the first 12 story actions are covered". A count is a position in
a list, and this list moves: delete an action from in front of the mark and
every later action slides down a slot, so the mark now covers one it has never
seen. Every rule in `memorybank` about sliding cursors, rewinding them and
translating between positions and `Action.index` existed to patch that up, and
each was a separate chance to get it wrong in a way nothing reports.
Two things are built from the story and stored beside it: the memories and the
story summary. Both need to record where they stopped.
A cursor here is an **anchor**: `(branch_id, depth)`, the node up to and
including which the work is done. Deleting an action does not move it, because
a depth is not a position — it is a coordinate along a path. "What is not
covered yet" becomes `history.count_after(anchor)`, which is a question about
the story rather than about a list index, and it answers correctly whatever has
been deleted from in front of it.
That mark used to be a count, such as "the first 12 story actions are covered".
A count is a position in a list, and this list changes. If you delete an action
in front of the mark, every later action moves down one slot, so the mark now
covers an action it never read. The rules in `memorybank` for sliding cursors,
rewinding them, and converting between positions and `Action.index` all existed
to correct for that, and each rule was a chance to introduce a silent error.
The branch half is what makes it survive forking. A depth alone is ambiguous
once two branches have a node 41; the anchor says which one, and
`Path.depth_on` reads it back as a depth on whatever story is being played —
capped at the fork, or "nothing covered" if the anchor sits on ground this path
never travelled. Until forking ships there is one branch and that is always a
no-op, which is the point: the coordinate system is right before anything needs
it to be.
A cursor here is an anchor instead. It stores `(branch_id, depth)`, naming the
node up to and including which the work is done. Deleting an action does not
move it, because a depth is a coordinate along a path rather than a position in
a list. The question "what is not covered yet" becomes
`history.count_after(anchor)`, which asks about the story rather than about a
list index, and it stays correct no matter what is deleted in front of it.
`NO_DEPTH` (-1) is "nothing covered", so a fresh adventure needs no special
case: every node is deeper than -1.
The branch half of the anchor is what makes it survive forking. A depth alone is
ambiguous once two branches both hold a node at depth 41. The anchor names the
branch, and `Path.depth_on` reads it back as a depth on whichever story is being
played. That read caps the depth at the fork, or reports nothing covered if the
anchor sits on a branch this path does not contain. Until forking ships there is
one branch, so this always returns the stored depth. That is the point: the
coordinate system is correct before anything depends on it.
`NO_DEPTH`, which is -1, means nothing is covered. A new adventure therefore
needs no special case, because every node is deeper than -1.
"""
from sqlalchemy.orm import Session
@@ -37,12 +38,11 @@ NO_DEPTH = lineage.NO_DEPTH
class Cursor:
"""One anchor on the adventure row: the memory bank's, or the summary's.
"""One anchor on the adventure row, for either the memory bank or the summary.
A pair of columns rather than a foreign key to the node. The node can be
deleted — that is most of what undo does — and the boundary is still
meaningful afterwards, so a pointer that has to resolve would be a pointer
that keeps not resolving.
The anchor is a pair of columns rather than a foreign key to the node. The
node can be deleted, which is what undo does, and the boundary still means
something afterwards. A foreign key would repeatedly fail to resolve.
"""
def __init__(self, name: str):
@@ -53,14 +53,14 @@ class Cursor:
# ------------------------------------------------------------- reading
def stored(self, adventure: models.Adventure) -> tuple[int | None, int]:
"""The anchor exactly as written, unread by any path."""
"""Returns the anchor as written, without resolving it against a path."""
depth = getattr(adventure, self.depth_field)
return getattr(adventure, self.branch_field), (
NO_DEPTH if depth is None else depth
)
def depth(self, db: Session, adventure: models.Adventure) -> int:
"""The anchor as a depth on the story currently being played."""
"""Returns the anchor as a depth on the story being played now."""
branch_id, depth = self.stored(adventure)
return lineage.path_of(db, adventure).depth_on(branch_id, depth)
@@ -69,44 +69,47 @@ class Cursor:
def anchor(
self, adventure: models.Adventure, branch_id: int | None, depth: int
) -> None:
"""Put the anchor at a coordinate given outright.
"""Sets the anchor to a coordinate supplied directly.
The plain setter under `anchor_at`. Only an import has a coordinate
without a node to read it off — a v2 bundle carries the anchor itself
(`app/bundle.py`), and the node it named lives in another database.
This is the plain setter beneath `anchor_at`. Only an import supplies a
coordinate with no node to read it from. A v2 bundle carries the anchor
itself, as described in `app/bundle.py`, and the node it named lives in
a different database.
"""
setattr(adventure, self.branch_field, branch_id)
setattr(adventure, self.depth_field, max(depth, NO_DEPTH))
def anchor_at(self, adventure: models.Adventure, node: models.Action) -> None:
"""Mark the work done up to and including `node`.
"""Marks the work done up to and including `node`.
Takes the node's own branch, not the adventure's head: a block of six
actions can end before the fork this branch was made at, and the
coverage belongs where the ground is.
This uses the node's own branch rather than the adventure's head. A block
of six actions can end before the fork that created the current branch,
and the coverage belongs where those actions are.
"""
self.anchor(adventure, node.branch_id, lineage.NO_DEPTH
if node.depth is None else node.depth)
def clear(self, adventure: models.Adventure) -> None:
"""Forget the anchor entirely: nothing is covered.
"""Clears the anchor, so that nothing counts as covered.
For when the ground the anchor stood on is gone — a deleted branch. On
Postgres a stale branch id would simply never resolve, but SQLite hands
a freed id to the next fork, and an anchor that resolves onto a branch
it has never seen would report a stretch of story as already
summarized. Clearing costs a re-summarize, which is the safe direction.
Call this when the branch the anchor referred to is deleted. On Postgres
a stale branch id would never resolve. On SQLite the next fork can reuse
an id that was just freed, and a stale anchor would then resolve onto a
branch it never saw and report that stretch of story as summarized.
Clearing the anchor costs one re-summarize, which is the safe direction
to be wrong in.
"""
self.anchor(adventure, None, NO_DEPTH)
def rewind_to(
self, adventure: models.Adventure, branch_id: int | None, depth: int
) -> None:
"""Move the anchor back to `depth` if it is past it; never forward.
"""Moves the anchor back to `depth` if it is past that depth.
The one direction that is safe without knowing what else has happened:
re-covering ground costs a summarizer call, skipping it loses a stretch
of story out of the memories for good.
The anchor never moves forward here. Moving backward is the only
direction that is safe without knowing what else changed. Covering
ground twice costs one summarizer call. Skipping ground removes a
stretch of story from the memories permanently.
"""
_, current = self.stored(adventure)
if current <= depth:
@@ -123,12 +126,12 @@ ALL = (MEMORY, SUMMARY)
def rewind_all(
adventure: models.Adventure, branch_id: int | None, depth: int
) -> None:
"""Hand a stretch of story back to *both* passes.
"""Returns a stretch of story to both the memory pass and the summary pass.
They move together because they cover the same ground from different sides:
the summary folds in the memories, so a memory withdrawn without rewinding
the summary leaves the summary claiming to have read something no longer
there.
The two move together because they cover the same actions from different
directions. The summary folds in the memories, so withdrawing a memory
without rewinding the summary would leave the summary claiming to have read
something that no longer exists.
"""
for cursor in ALL:
cursor.rewind_to(adventure, branch_id, depth)
@@ -137,18 +140,19 @@ def rewind_all(
def anchor_at_position(
adventure: models.Adventure, cursor: Cursor, position: int
) -> None:
"""Set `cursor` from a count of covered story actions — a v1 bundle's mark,
or a database written before the anchors existed.
"""Sets `cursor` from a count of covered story actions.
The position-th story action in depth order is the node that says the same
thing, and goes on saying it once something in front of it is deleted. A
position past the end of the story is not a bad value: an adventure caught
up under the older rule can carry one, and it means the same thing the tip
does, so that is where it lands.
A v1 bundle stores its mark as a count, and so does a database written
before the anchors existed.
The SQL half of this rule is `migrations._backfill_cursor_anchors`, which
has to do it for every adventure at once without loading any of them; the
two must agree.
The action at `position` in depth order is the node that carries the same
meaning, and it keeps that meaning after something in front of it is
deleted. A position past the end of the story is not an invalid value. An
adventure that was fully caught up under the old rule can hold one, and it
means the same thing as the tip, so this function anchors at the tip.
`migrations._backfill_cursor_anchors` implements this rule in SQL for every
adventure at once, without loading any of them. The two must agree.
"""
if position <= 0:
return
@@ -158,11 +162,11 @@ def anchor_at_position(
def position_of(adventure: models.Adventure, depth: int) -> int:
"""How many story actions lie at or before `depth` — an anchor read back as
a count.
"""Returns how many story actions lie at or before `depth`.
The v1 export bundle stores the cursors as positions, and a v1 bundle is
read by builds that have never heard of a depth. This is the one place that
still speaks that coordinate system, and SP6's v2 format retires it.
This reads an anchor back as a count. The v1 export bundle stores cursors as
counts, and builds that have never used depths read v1 bundles. This
function is the only remaining code that speaks that coordinate system. The
v2 format introduced in SP6 replaces it.
"""
return max(history.count(adventure) - history.count_after(adventure, depth), 0)
backend/app/context/history.py
+140 -124
View File
@@ -1,42 +1,44 @@
"""Reading the story without reading all of it.
"""Reads part of a story without loading all of it.
`story_actions()` walked `adventure.actions`, which loads every row of the
adventure — then every caller threw almost all of it away. The context builder
concatenates the story and immediately cuts it back to the token budget; the
NPC-in-scene check looks at the last 6; memory retrieval looks at the last 4;
the post-turn cursor clamp only wants a count. So a turn on a 200-action
adventure read ~840 KB to use maybe 70 KB of it, and the cost grew with every
turn played.
`story_actions()` used to walk `adventure.actions`, which loads every row of the
adventure. Every caller then discarded nearly all of those rows. The context
builder joins the story and immediately trims it to the token budget. The
in-scene NPC check reads the last 6 actions. Memory retrieval reads the last 4.
The post-turn cursor clamp needs only a count. A turn on a 200-action adventure
read about 840 KB in order to use about 70 KB, and the cost grew with every
turn.
This module serves those shapes directly from SQL — a tail, a slice, a count —
so the read is bounded by the context budget instead of by the length of the
story.
This module serves those shapes from SQL directly, as a tail, a slice, or a
count. A read is therefore bounded by the context budget rather than by the
length of the story.
Three rules hold everything together:
Three rules hold the module together:
* **One definition of "story action".** Membership decides what a reader sees
and what the summarizer is handed, so SQL and Python must agree on it
exactly. `_STORY_TEXT` and `is_story_text()` are that one definition, written
twice; keep them in step.
* **Never load twice.** If `adventure.actions` is already in memory (the
scripting pipeline hands the whole history to user scripts, as AI Dungeon
does), every helper here slices that instead of issuing a query, so a
scripted adventure pays what it always paid and nothing more.
* **Every read goes through the branch clause** (Phase 14). `adventure.actions`
is every branch's actions, not the story being played — so the shortcut above
cuts the loaded collection down to the path before slicing it, exactly as the
SQL does. This is the line that would silently assemble a prompt out of two
different stories, which is why `lineage.Path` owns both halves of it.
- There is one definition of a story action. That definition decides both what
a reader sees and what the summarizer receives, so the SQL and the Python must
agree exactly. `_STORY_TEXT` and `is_story_text()` express the same rule
twice. Keep them in step.
- No caller loads the same rows twice. If `adventure.actions` is already in
memory, every helper here slices that collection instead of running a query.
The scripting pipeline hands the whole history to user scripts, as AI Dungeon
does, so a scripted adventure costs no more than it did before.
- Every read applies the branch clause, as of Phase 14. `adventure.actions`
holds the actions of every branch rather than the story being played, so the
in-memory path filters the collection down to the path before slicing it, in
the same way the SQL does. Skipping that filter would build a prompt from two
different stories without reporting an error, which is why `lineage.Path`
owns both forms of the rule.
Ordering is by `depth` now, not `index`. Since SP4 the two can hold the same
number on *different rows* — attempts at one turn share both — so only `depth`
plus the branch clause's `live` test says which of them the story is.
Reads order by `depth` rather than `index`. Since SP4, two different rows can
hold the same value for both columns, because the attempts at one turn share
them. Only `depth` together with the `live` test in the branch clause
identifies the row the story uses.
SP3 added the reads that count *from a node* rather than from the start —
`count_after`, `after`, `newest`. The memory bank used to ask for
"positions 12 to 18 of the story", which is a question whose answer moves when
an action is deleted from in front of it. It now asks for "the six actions
after depth 41", which is the same question a fork has to answer anyway.
SP3 added the reads that count from a node rather than from the start:
`count_after`, `after`, and `newest`. The memory bank used to ask for positions
12 through 18 of the story, and the answer to that question changes when an
action in front of those positions is deleted. It now asks for the six actions
after depth 41, which is the question that forking requires in any case.
"""
from sqlalchemy import func, inspect as sa_inspect
@@ -45,17 +47,17 @@ from sqlalchemy.orm import Session, defer, object_session
from .. import models
from . import lineage
# How many of the newest actions to read before checking whether the token
# budget is covered. When it isn't, the next size is worked out from the
# average action length just measured rather than by blind doubling — guessing
# high means reading hundreds of actions to use sixty of them.
# How many of the newest actions to read before checking whether they cover the
# token budget. If they do not, the next size comes from the average action
# length just measured rather than from doubling the previous size. Doubling
# overshoots, which means reading hundreds of actions in order to use sixty.
WINDOW_START = 32
WINDOW_MARGIN = 0.15 # aim this far past the budget, so one more round is rare
WINDOW_STEP = 8 # ...and at least this many more actions each round
WINDOW_MARGIN = 0.15 # Aim this far past the budget, so a second round is rare.
WINDOW_STEP = 8 # Read at least this many more actions in each round.
# Depth is the ordering key; id breaks the tie that a pre-tree row (depth NULL,
# and so invisible anyway) or a future sibling pair would otherwise leave to
# the database's mood.
# `depth` is the ordering key, and `id` breaks ties. Without `id`, the database
# would choose the order. Two rows can share a depth: a pre-tree row, which has
# a NULL depth and is invisible to reads, or a pair of sibling attempts.
_OLDEST_FIRST = (models.Action.depth, models.Action.id)
_NEWEST_FIRST = (models.Action.depth.desc(), models.Action.id.desc())
@@ -63,12 +65,13 @@ _NEWEST_FIRST = (models.Action.depth.desc(), models.Action.id.desc())
def _sql_stripped(column):
"""`column` with leading/trailing whitespace removed, portably.
SQLite and Postgres both accept single-argument `trim()`, but it strips
spaces only — Python's `.strip()` also drops newlines and tabs, and an
action of nothing but a newline would otherwise count as story text here
and not in Python. `replace()` and `trim()` are the two string functions
both dialects spell identically, so fold the other whitespace into spaces
first. (Form feed and vertical tab are not covered; nothing produces them.)
SQLite and Postgres both accept `trim()` with a single argument, but that
form removes spaces only. Python's `str.strip()` also removes newlines and
tabs. Without this helper, an action containing only a newline would count
as story text in SQL but not in Python. Both dialects spell `replace()` and
`trim()` the same way, so this function converts the other whitespace to
spaces first. It does not handle form feed or vertical tab, because nothing
produces them.
"""
folded = column
for char in ("\n", "\r", "\t"):
@@ -80,15 +83,18 @@ _STORY_TEXT = _sql_stripped(models.Action.text) != ""
def is_story_text(text: str) -> bool:
"""The Python half of `_STORY_TEXT` — keep the two in step."""
"""Returns whether `text` counts as story text.
This is the Python form of `_STORY_TEXT`. Keep the two in step.
"""
return bool(text.strip())
def _loaded_actions(adventure: models.Adventure) -> list[models.Action] | None:
"""The adventure's actions if they are already in memory, else None.
Slicing an already-loaded collection is free; issuing a query beside it
would mean paying for the same rows twice.
Slicing a collection that is already loaded costs nothing, and running a
query beside it would fetch the same rows a second time.
"""
state = sa_inspect(adventure)
if state.detached or "actions" in state.unloaded:
@@ -101,11 +107,14 @@ def _from_memory(
) -> list[models.Action] | None:
"""The story, from the already-loaded collection, or None to go to SQL.
The collection is the *adventure's* actions — every branch of it. Cutting
it down to the path here is the same filter the SQL applies, and skipping
it would hand the context builder a prompt assembled from siblings of the
story being played. The path needs a session to read the branch row from;
without one there is no answer to give, so say so rather than guess.
The collection holds the adventure's actions, which means the actions of
every branch. Filtering it down to the path here applies the same rule that
the SQL applies. Without that filter, the context builder would receive a
prompt built from siblings of the story being played.
Resolving the path requires a session to read the branch row from. If there
is no session, this function returns None so that the caller falls back to
SQL rather than guessing.
"""
loaded = _loaded_actions(adventure)
if loaded is None:
@@ -131,11 +140,10 @@ def _filters(
exclude_action_id: int | None,
entries: int | None = None,
) -> list:
# adventure_id is redundant beside the branch clause — branch ids are
# unique, so a branch already names one adventure. It stays because it is
# the cheap half of the check that catches a node written onto the wrong
# adventure's branch, and because a clause nobody can read is a clause
# nobody maintains.
# `adventure_id` is redundant beside the branch clause, because branch ids
# are unique and a branch already identifies one adventure. The filter
# remains because it costs little, it catches a node written onto another
# adventure's branch, and it makes the query easier to read.
conditions = [
models.Action.adventure_id == adventure.id,
path.clause(models.Action, count=entries),
@@ -171,11 +179,11 @@ def _count_query(
):
"""A real `SELECT count(...)`.
Deliberately not `_query(...).count()`: that wraps the entity select in a
subquery, so the emitted SQL names every column — including the deferred
ones this whole design exists to keep off the wire. No bytes come back
either way, but the database still has to read them, and an egress guard
that greps the SQL cannot tell the two apart.
This function deliberately avoids `_query(...).count()`. That form wraps the
entity select in a subquery, so the emitted SQL names every column,
including the deferred columns that this design keeps off the wire. Neither
form returns those bytes to the client, but the database still reads them,
and an egress guard that inspects the SQL cannot tell the two forms apart.
"""
return db.query(func.count(models.Action.id)).filter(
*_filters(adventure, path, exclude_action_id, entries)
@@ -197,14 +205,14 @@ def story_actions(
) -> list[models.Action]:
"""Every story action, oldest first.
Still the right call where the whole story is genuinely wanted — user
scripts receive it, per AI Dungeon's scripting API. Prefer `tail`, `slice_`
or `count` anywhere the caller only needs part of it.
Call this function when you need the whole story. User scripts receive it,
which matches AI Dungeon's scripting API. Use `tail`, `slice_`, or `count`
when you need only part of the story.
`exclude_action_id` drops one action from the story — used by retry, where
the attempt being replaced is still the live node of its turn (it stays
live until a replacement exists) but must not appear in the context
assembled to replace it.
`exclude_action_id` removes one action from the result. Retry uses it. The
attempt being replaced is still the live node of its turn, because it stays
live until a replacement exists, but it must not appear in the context that
is assembled to replace it.
"""
in_memory = _from_memory(adventure, exclude_action_id)
if in_memory is not None:
@@ -241,18 +249,20 @@ def tail_range(
) -> list[models.Action]:
"""`limit` story actions ending `skip` actions before the end, oldest first.
`skip=0` is the newest slice; `skip=32, limit=16` is the 16 actions just
older than the newest 32. Lets a growing window fetch only the part it
doesn't already have.
Passing `skip=0` returns the newest slice. Passing `skip=32` and `limit=16`
returns the 16 actions immediately older than the newest 32. A growing
window therefore fetches only the actions it does not already hold.
This is the read the lineage window exists for. The path's ranges are
disjoint and descending, so the newest N nodes come from the newest few
lineage entries and the rest of the ancestry need not be named at all: a
story forked two hundred times reads its tail with as few clauses as one
forked never. `prefix_covering` estimates how many entries that takes from
depth arithmetic alone; the estimate is only ever short where a middle
action was deleted, and then the read widens to the whole lineage and pays
one more query.
This read is the reason the lineage window exists. The path's ranges do not
overlap and they descend, so the newest N nodes come from the newest few
lineage entries and the query never has to name the rest of the ancestry. A
story that has forked 200 times reads its tail with as few clauses as one
that has never forked.
`prefix_covering` estimates how many entries that takes, using depth
arithmetic alone. The estimate falls short only when an action was deleted
from the middle of the story. In that case this function widens the read to
the whole lineage, at the cost of one more query.
"""
if limit <= 0 or skip < 0:
return []
@@ -295,11 +305,12 @@ def slice_(
) -> list[models.Action]:
"""Story actions at positions [start, start + length), oldest first.
Positions are into the same filtered, depth-ordered list the memory cursors
count in, which is why the filter has to match Python's exactly.
Positions index into the same filtered, depth-ordered list that the memory
cursors count in, which is why the SQL filter must match the Python filter
exactly.
Counts from the oldest end, so it names the whole lineage: there is no
prefix of the ancestry that holds "the story's first ten actions".
This function counts from the oldest end, so it names the whole lineage. No
prefix of the ancestry contains the first ten actions of the story.
"""
if length <= 0 or start < 0:
return []
@@ -321,9 +332,9 @@ def slice_(
def depth_of(action: models.Action) -> int:
"""`action.depth`, with the no-depth case spelled once.
A row with no depth is a pre-tree row, which no path contains — so it can
only turn up in an already-loaded collection, and it sorts before the story
rather than after it.
A row with no depth predates the tree. No path contains such a row, so it
appears only in a collection that is already loaded. It sorts before the
story rather than after it.
"""
return action.depth if action.depth is not None else lineage.NO_DEPTH
@@ -333,14 +344,14 @@ def count_after(
) -> int:
"""How many story actions lie past `depth` on the path.
The node-anchored replacement for "the story is N long and the cursor is at
M". Deleting an action from in front of the boundary makes this number
smaller, which is true; it does not make the boundary point somewhere else,
which is the bug the positions had.
This replaces the older calculation, which compared the length of the story
with the position of the cursor. Deleting an action in front of the boundary
makes this number smaller, which is correct. It does not move the boundary
to a different action, which is the error that positions produced.
`covering_after` says exactly which lineage entries can hold a node deeper
than the boundary, so a cursor near the tip names one branch however many
forks are below it.
`covering_after` reports which lineage entries can hold a node deeper than
the boundary, so a cursor near the tip names one branch however many forks
lie below it.
"""
in_memory = _from_memory(adventure, exclude_action_id)
if in_memory is not None:
@@ -367,8 +378,9 @@ def after(
) -> list[models.Action]:
"""The oldest `limit` story actions past `depth`, oldest first.
"The next block the summarizer has not seen", asked as a fact about the
story rather than as an offset into a list that shifts underneath it.
This returns the next block that the summarizer has not read. It asks a
question about the story rather than using an offset into a list whose
entries move.
"""
if limit <= 0:
return []
@@ -391,14 +403,15 @@ def after(
def newest(adventure: models.Adventure) -> models.Action | None:
"""The newest story action, or None on an empty story.
A row, not a count and an offset: this is the node an anchor moves to when
derived work catches up with the end of the story.
This returns a row rather than a count and an offset, because it names the
node an anchor moves to when derived work reaches the end of the story.
It used to be the *second* newest — the memory bank held one action back
because retry rewrote a row, so a memory covering the newest action could
end up describing narration the player had retried away. Since SP4 a retry
writes a sibling instead, and the coordinate's derived work is withdrawn
when the story at it changes, so there is nothing left to hold back.
It used to return the second newest action. The memory bank held one action
back because retry rewrote a row, so a memory that covered the newest action
could describe narration the player had already replaced. Since SP4, a retry
writes a sibling row instead, and the derived work at a coordinate is
withdrawn when the story at that coordinate changes. There is nothing left
to hold back.
"""
rows = tail(adventure, 1)
return rows[0] if rows else None
@@ -407,11 +420,12 @@ def newest(adventure: models.Adventure) -> models.Action | None:
def max_action_index(adventure: models.Adventure) -> int:
"""Highest `Action.index` in the adventure, story text or not. -1 if empty.
The one read here that is deliberately *not* path-scoped. `index` is the
legacy column, kept unread until SP8 drops it, and its only remaining job
is to hand the next row a number nothing else holds — which is a fact about
the adventure, not about the story being played. Scoping it to a branch
would let two branches issue the same index.
This is the only read in the module that is deliberately not scoped to a
path. `index` is a legacy column that remains unread until SP8 drops it. Its
one remaining job is to give the next row a number that no other row holds,
which is a fact about the adventure rather than about the story being
played. Scoping the query to a branch would let two branches issue the same
index.
"""
loaded = _loaded_actions(adventure)
if loaded is not None:
@@ -433,16 +447,18 @@ def window_covering(
token_counter,
exclude_action_id: int | None = None,
) -> list[models.Action]:
"""The newest story actions whose combined text exceeds `budget_tokens` —
i.e. more than the context builder can possibly include, and never less.
"""Returns the newest story actions whose combined text exceeds `budget_tokens`.
Measures rather than guesses a chars-per-token ratio, so the prompt is
byte-for-byte what loading the whole story would have produced. Budgets on
the raw text, which is never longer than the rendered history text, so
erring here can only mean fetching slightly too much.
The result always holds at least as much text as the context builder can
include, and never less.
Each round fetches only the actions it doesn't already hold, so no row is
ever read twice however many rounds it takes.
This function counts tokens rather than estimating a characters-per-token
ratio, so the prompt matches what loading the whole story would produce. It
budgets against the raw text, which is never longer than the rendered
history text, so any error causes it to fetch slightly more than needed.
Each round fetches only the actions it does not already hold, so no row is
read twice however many rounds the loop takes.
"""
actions: list[models.Action] = []
tokens = 0
@@ -452,15 +468,15 @@ def window_covering(
adventure, len(actions), size - len(actions), exclude_action_id
)
if not older:
return actions # already holding the whole story
return actions # The result already holds the whole story.
actions = older + actions
tokens += sum(token_counter(a.text) for a in older)
if len(actions) < size:
return actions # that was the whole story
return actions # That was the whole story.
if tokens > budget_tokens:
return actions
# Short. Project how many actions the budget takes at the length these
# ones turned out to be, and go straight there.
# The window is still short. Estimate how many actions the budget needs
# at the average length just measured, then read that many.
average = tokens / len(actions)
projected = int(budget_tokens / average * (1 + WINDOW_MARGIN)) + WINDOW_STEP
size = max(projected, size + WINDOW_STEP)
backend/app/context/lineage.py
+116 -106
View File
@@ -1,35 +1,35 @@
"""Phase 14 — which nodes are "this story".
"""Phase 14: decides which nodes make up one story.
`tree.py` decides where a node is written. This module is the other half: it
decides which nodes a read can see, and it is the **only** place that knows.
`tree.py` decides where a node is written. This module decides which nodes a
read can see, and it is the only place that makes that decision.
A branch owns the nodes played on it and *borrows* everything before its fork
point from its ancestors, so "the story on branch C" is not a column you can
filter on — it is an OR of ranges::
A branch owns the nodes played on it and inherits everything before its fork
point from its ancestors. "The story on branch C" is therefore not a value you
can filter a column on. It is an OR of ranges::
(branch_id = C) -- C's own nodes, to the tip
(branch_id = C) -- C's own nodes, through to the tip
OR (branch_id = B AND depth <= 5)
OR (branch_id = A AND depth <= 3)
which is exactly what `branches.lineage` spells out, newest first, computed
once when the fork happens. Reads never walk parent pointers to rebuild it.
The `branches.lineage` column records exactly that list, newest first. The fork
computes it once, so no read walks parent pointers to rebuild it.
Two properties fall out of the shape, and both are load-bearing:
The shape of the list gives two properties that the windowed reads depend on:
* **The ranges are disjoint and descending.** A branch's own nodes always sit
deeper than its fork point, and each lineage entry is capped at the fork
depth of the branch beneath it. So ordering the whole clause by `depth`
descending is the same as reading entry 0's nodes, then entry 1's, then
entry 2's — which is what lets a tail read use only the newest few entries
and stop.
* **Clause count is bounded by the context window, not by fork count.** A
200-fork story whose newest branch is 40 turns long reads with one clause,
because the window is covered before the second entry is reached. That is
`prefix_covering`, and it is why `history.window_covering` can keep its shape.
- The ranges do not overlap, and they descend. A branch's own nodes always sit
deeper than its fork point, and each lineage entry is capped at the fork depth
of the branch below it. Ordering the whole clause by `depth` descending
therefore reads entry 0's nodes, then entry 1's, then entry 2's. A tail read
can use the newest few entries and stop.
- The number of clauses depends on the size of the context window, not on how
many times the story has forked. A story with 200 forks whose newest branch
runs 40 turns reads with a single clause, because the window is covered before
the second entry is reached. `prefix_covering` implements this, and it is why
`history.window_covering` can keep its current shape.
Everything here is a read. Nothing in this module creates a branch or writes a
row: an adventure with no branch has no story, and healing that is the write
side's job (`tree.place_action`, and the flush guard in `models.py` behind it).
Everything in this module reads. Nothing here creates a branch or writes a row.
An adventure with no branch has no story, and the write side repairs that. See
`tree.place_action` and the flush listener in `models.py`.
"""
from sqlalchemy import and_, false, or_
@@ -37,31 +37,33 @@ from sqlalchemy.orm import Session
from .. import models
# The depth of an adventure with no actions. Mirrors tree.NO_DEPTH; kept
# separately so a read never has to import the write half.
# The depth of an adventure that has no actions. This mirrors `tree.NO_DEPTH`.
# It is duplicated here so that a read never has to import the write half.
NO_DEPTH = -1
# The opening node of an adventure. Depth 0 exists only on the root branch — a
# fork starts its own nodes after the depth it forked at — so this names one
# node per adventure, not one per branch. It is also where migration 62 parked
# every memory written before memories had coordinates, which is why the two
# places that can retire a memory (`memorybank.forget_node`, and a v1 import
# with no depth to read) both have to say something about it.
# The opening node of an adventure. Depth 0 exists only on the root branch,
# because a fork starts its own nodes after the depth it forked at. This
# constant therefore names one node per adventure rather than one per branch.
#
# Migration 62 also placed every memory written before memories had coordinates
# at depth 0. That is why both places that can retire a memory must handle this
# depth: `memorybank.forget_node`, and a v1 import that has no depth to read.
ROOT_DEPTH = 0
def entries_of(branch: models.Branch) -> list[tuple[int, int | None]]:
"""`branch.lineage` as (branch_id, max_depth) pairs, newest first.
"""Returns `branch.lineage` as (branch_id, max_depth) pairs, newest first.
An empty lineage reads as "this branch alone, to its tip" rather than as an
error. That is what a root branch's lineage means, and it is what a branch
row looks like in the moment between being inserted and having its own id
to name — so the fallback is the truth, not a guess.
An empty lineage means this branch alone, through to its tip. That is not an
error. It is what a root branch's lineage means, and it is also what a
branch row holds between the moment it is inserted and the moment its own id
is written into the column. The fallback is therefore correct rather than a
guess.
"""
raw = branch.lineage if isinstance(branch.lineage, list) else []
entries: list[tuple[int, int | None]] = []
for item in raw:
# JSON round-trips lists; a hand-written row might hold tuples.
# JSON round-trips lists, but a hand-written row might hold tuples.
if not isinstance(item, (list, tuple)) or not item:
continue
branch_id = item[0]
@@ -73,10 +75,10 @@ def entries_of(branch: models.Branch) -> list[tuple[int, int | None]]:
class Path:
"""One story, as a clause and as a predicate.
"""One story, expressed as a SQL clause and as a Python predicate.
Holds the lineage entries newest first, plus the depth of the tip, which is
only used to estimate how much story each entry covers.
The object holds the lineage entries newest first, plus the depth of the
tip. The tip is used only to estimate how much story each entry covers.
"""
def __init__(self, entries: list[tuple[int, int | None]], tip: int | None = None):
@@ -96,28 +98,29 @@ class Path:
model=models.Action,
count: int | None = None,
):
"""The branch clause, over `model` (`Action` or `Memory`).
"""Returns the branch clause over `model`, which is `Action` or `Memory`.
`count` limits it to the newest `count` lineage entries — the windowed
read. `None` is the whole lineage, which is what anything counting from
the *oldest* end (a slice, a total) has to use.
`count` limits the clause to the newest `count` lineage entries, which
produces a windowed read. Pass `None` for the whole lineage. Any caller
that counts from the oldest end, such as a slice or a total, must pass
`None`.
Every row this reads has a depth. Memories used to be the exception —
a hand-written one had a branch and no depth, and needed an escape
clause here to survive being capped at a fork. SP7 anchors them at the
head instead (`tree.place_memory`), which is a better answer to the same
problem: the memory is not exempt from the path, it is *on* one. A row
with no depth is now a pre-tree leftover that no read should see.
Every row this clause selects has a depth. Memories were once an
exception, because a hand-written memory had a branch but no depth and
needed an escape clause here to avoid being capped at a fork. SP7
anchors those memories at the head instead. See `tree.place_memory`. A
memory is now on a path rather than exempt from one, and a row with no
depth is a pre-tree leftover that no read should return.
Actions also have to be *live* (SP4). A coordinate can hold several
attempts at the same turn, and the story tells one of them; the losing
siblings sit at the same branch and depth and are excluded here, once,
so that no read of the story has to know that retries exist. Only
`app/attempts.py` looks past this.
Actions must also be live, as of SP4. One coordinate can hold several
attempts at a turn, and the story uses one of them. The other attempts
sit at the same branch and depth, and this clause excludes them once, so
that no read of the story has to account for retries. Only
`app/attempts.py` looks past this filter.
An empty path yields `false`, not "no filter": an adventure whose nodes
carry no branch has no story, and the loud version of that is an empty
page, not every branch at once.
An empty path returns `false` rather than no filter at all. An adventure
whose nodes carry no branch has no story, and the correct way to show
that is an empty page rather than every branch at once.
"""
entries = self.entries if count is None else self.entries[:count]
if not entries:
@@ -136,14 +139,16 @@ class Path:
# ------------------------------------------------------------- Python
def contains(self, node) -> bool:
"""The Python half of `clause()` — keep the two in step.
"""Returns whether `node` is on this path.
Used where the rows are already in memory (the scripting pipeline hands
user scripts the whole history), so an already-loaded collection can be
cut down to the path without a second read.
This is the Python equivalent of `clause()`. Keep the two in step.
`live` is checked first, and only on rows that have the attribute:
memories have no siblings to lose to.
Callers use it where the rows are already in memory. The scripting
pipeline hands user scripts the whole history, so a loaded collection
can be reduced to the path without a second query.
The method checks `live` first, and only on rows that define the
attribute, because memories have no siblings.
"""
if getattr(node, "live", True) is False:
return False
@@ -157,22 +162,25 @@ class Path:
return False
def sort_key(self, node) -> tuple[int, int]:
"""Oldest-first ordering. `depth` is the ordering key; `id` breaks the
tie a pre-tree row (depth NULL) or a future sibling pair would leave."""
"""Returns a sort key that orders nodes from oldest to newest.
`depth` is the ordering key. `id` breaks ties, which a pre-tree row with
a NULL depth or a pair of siblings can produce.
"""
return (node.depth if node.depth is not None else NO_DEPTH, node.id or 0)
# ------------------------------------------------------------ windowing
def prefix_covering(self, rows: int) -> int:
"""How many lineage entries it takes to hold the newest `rows` nodes.
"""Returns how many lineage entries hold the newest `rows` nodes.
An estimate from depth arithmetic, not a query: entry *i* covers the
depths between its own cap and the cap of the entry below it, and there
is at most one node per depth on a path. So the count it returns is
never too many, and is too few only where the story has gaps — an
action deleted from the middle. The caller widens to the full lineage
if the window comes up short, which costs a second query on a story
somebody has deleted from, and nothing at all otherwise.
The result is an estimate from depth arithmetic rather than a query.
Entry *i* covers the depths between its own cap and the cap of the entry
below it, and a path holds at most one node per depth. The estimate is
therefore never too large. It is too small only when the story has gaps,
which happens after an action is deleted from the middle. In that case
the caller widens the read to the whole lineage, which costs one extra
query.
"""
total = len(self.entries)
if rows <= 0 or total == 0:
@@ -182,9 +190,9 @@ class Path:
top = self.tip if max_depth is None else max_depth
below = self.entries[i + 1][1] if i + 1 < total else NO_DEPTH
if top is None or below is None:
# No tip recorded, or a cap missing from a hand-written row:
# nothing to estimate from, so read the lot rather than guess
# short and hide the older half of the story.
# Either no tip was recorded, or a hand-written row is missing a
# cap. There is nothing to estimate from, so return every entry.
# Guessing low would hide the older half of the story.
return total
covered += max(top - below, 0)
if covered >= rows:
@@ -192,15 +200,15 @@ class Path:
return total
def covering_after(self, depth: int) -> int:
"""How many lineage entries can hold a node deeper than `depth`.
"""Returns how many lineage entries can hold a node deeper than `depth`.
The counterpart to `prefix_covering`, and unlike it this is exact
rather than an estimate: entry *i* holds nothing deeper than its own
cap, and the caps descend, so the first entry capped at or below
`depth` ends the search — it and everything older is behind the
boundary. Reading "the story after the cursor" therefore names one
branch on any story whose cursor is on its newest branch, however
often it has forked.
This is the counterpart to `prefix_covering`, and it is exact rather than
an estimate. Entry *i* holds nothing deeper than its own cap, and the
caps descend, so the first entry capped at or below `depth` ends the
search. That entry and every older one fall behind the boundary. As a
result, reading the story after the cursor touches one branch on any
story whose cursor sits on its newest branch, however many times the
story has forked.
"""
for i, (_, max_depth) in enumerate(self.entries):
if max_depth is not None and max_depth <= depth:
@@ -208,27 +216,28 @@ class Path:
return len(self.entries)
def depth_on(self, branch_id: int | None, depth: int) -> int:
"""A stored `(branch_id, depth)` anchor, read as a depth on *this* path.
"""Reads a stored `(branch_id, depth)` anchor as a depth on this path.
An anchor is how far along a story some derived work has got — which
memories cover, what the summary has folded in. It names a node, so
moving to another path has to be answered rather than assumed:
An anchor records how far along a story some derived work reached, such
as which actions the memories cover or what the summary folded in. The
anchor names a node, so moving to a different path needs an explicit
answer. There are two cases:
* the anchor's branch is on this path — the depth stands, capped at the
fork the path takes off that branch, because nothing past the fork is
on this story;
* the branch is not on this path at all — the work was done on ground
this story never travelled, so nothing here is covered.
- The anchor's branch is on this path. The depth stands, capped at the
fork where this path leaves that branch, because nothing past the fork
belongs to this story.
- The anchor's branch is not on this path. The work was done on a branch
this story does not contain, so nothing here counts as covered.
The second case cannot arise while an adventure has one branch: the
anchor is always set from a node on it. It exists because the fallback
for "I don't know" must be to redo the work, not to skip it.
The second case cannot occur while an adventure has one branch, because
the anchor is always set from a node on it. It exists because the safe
answer to an unknown anchor is to redo the work rather than skip it.
"""
if depth <= NO_DEPTH:
return NO_DEPTH
if branch_id is None:
# A pre-tree anchor, or one set by hand. There is one story, so the
# depth is a position in it and means what it says.
# The anchor predates the tree, or someone set it by hand. There is
# only one story, so the depth is a position in it.
return depth
for entry_branch, max_depth in self.entries:
if entry_branch == branch_id:
@@ -237,18 +246,19 @@ class Path:
def branch_of(db: Session, adventure: models.Adventure) -> models.Branch | None:
"""The branch this adventure is being read at, or None if it has none.
"""Returns the branch this adventure is read at, or None if it has none.
Deliberately not `tree.head_branch`, which creates one: a GET must not
write. An adventure with no branch row also has no nodes carrying a branch,
so the two agree — both say "no story here".
This function is deliberately not `tree.head_branch`, which creates a branch.
A GET request must not write. An adventure with no branch row also has no
nodes that carry a branch, so both answers agree that there is no story.
"""
if adventure.head_branch_id is not None:
branch = db.get(models.Branch, adventure.head_branch_id)
if branch is not None:
return branch
# A head naming a branch that is gone: fall through to the root, the
# same recovery `tree.head_branch` makes on the write side.
# The head points at a branch that no longer exists. Fall through to the
# root, which is the same recovery that `tree.head_branch` performs on
# the write side.
return (
db.query(models.Branch)
.filter(
@@ -261,7 +271,7 @@ def branch_of(db: Session, adventure: models.Adventure) -> models.Branch | None:
def path_of(db: Session, adventure: models.Adventure) -> Path:
"""The story the adventure's head is currently on."""
"""Returns the story that the adventure's head currently sits on."""
branch = branch_of(db, adventure)
if branch is None:
return Path([], adventure.head_depth)