interactive-story/backend/app/knowledge/fts.py

"""M7: the SQLite FTS5 lexical index over imported passages.

Lexical retrieval is a **supported production path**, not a fallback for when
the embeddings are broken. It is the half that finds `Old Abbey`,
`broken-circle` and `Westhaven` — proper nouns and invented terms, which is most
of what a setting bible is made of and precisely what an embedding trained on
ordinary English is worst at. `IMPORTED-KNOWLEDGE-DESIGN.md` §24 chooses FTS5
for being transparent, fast and deterministic, and §23 requires it to keep
working when the semantic side does not.

## The table

    CREATE VIRTUAL TABLE knowledge_fts USING fts5(text, tokenize='porter unicode61')

One column, and `rowid` is the chunk's primary key. Everything else — which
campaign, which source, whether that source is enabled — is on
`knowledge_chunks` and `knowledge_sources`, and the search below joins to them.
That is deliberate: the scope rules are then enforced by the same rows the rest
of the application reads, rather than by a copy inside the index that could
drift out of step with them.

`text` is the heading trail and the body together. A heading is a strong signal
and often the only place a term appears — "Old Abbey" is a heading in the
standard fixture, not a sentence in it — so indexing the body alone would miss
the exact query the acceptance test asks.

A virtual table is not something `Base.metadata.create_all` can build, so this
module owns its DDL and `migrations.bootstrap` calls `ensure`.

## Why not `content=` external-content mode

External content would save storing the passage text twice. It also makes every
delete a three-way ceremony (`INSERT INTO t(t, rowid, text) VALUES('delete',...)`)
that must be handed the *old* text, and a mismatch corrupts the index silently
rather than raising. Sources here are capped at a megabyte and a campaign holds
a handful, so the duplicate text is worth an index whose delete is `DELETE`.
"""

from __future__ import annotations

import re

from sqlalchemy import text as sql
from sqlalchemy.orm import Session

TABLE = "knowledge_fts"

# `porter unicode61` — Unicode-aware tokenizing with English stemming on top.
#
# Stemming is what makes the lexical half work on prose written by a person who
# was not thinking about the index. A reader asks about "resurrecting" Edrin and
# the Canon file says "resurrection"; a scene mentions "gates" and the source
# says "gate". Without a stemmer those are misses, and the reader has no way to
# know why — which would make lexical retrieval a keyword game rather than the
# production path it is meant to be.
#
# It costs nothing on the terms that matter most. Porter only strips recognised
# English suffixes, so `Westhaven`, `Mara` and `broken-circle` are unchanged,
# and the query is stemmed by the same rule as the index, so the two always
# agree. The alternative, plain `unicode61`, was measured failing the ordinary
# case above.
DDL = (
    f"CREATE VIRTUAL TABLE IF NOT EXISTS {TABLE} "
    "USING fts5(text, tokenize='porter unicode61')"
)

# Everything FTS5 reads as syntax rather than as a word. The query builder below
# never passes these through: each term is wrapped in double quotes, which makes
# it a literal phrase, and any quote inside it is doubled. So a source or a
# scene containing `NEAR(` or `*` or `"` produces a search for those characters
# rather than a malformed query or an operator the caller did not ask for.
_TERM_SPLIT = re.compile(r"[^\w'\-]+", re.UNICODE)
# Words too common to be evidence of anything.
#
# This list is deliberately limited to **function words and contentless
# generics**. It does not contain a single word about taverns, abbeys, keys or
# any other subject, because a stop list that starts removing subject matter is
# how a search stops finding "The Silver Key".
#
# It was widened in the M7 corrective pass. The original 42 words let a passage
# be admitted into an orbital-mechanics scene on the word **"before"** — one
# generic token was enough, because nothing downstream asked how much had
# actually matched (review finding M7-F1). Both halves of that were wrong and
# both are fixed: the word is filtered here, and `classes.LEXICAL_MIN_TERMS`
# now requires more than one term anyway.
_STOP = frozenset("""
a about above after again against all almost along already also although always
am among an and another any anyone anything are around as at
back be became because become been before began begin behind being below beside
best better between beyond both bring but by
came can cannot could
did do does doing done down during
each either else enough even ever every everyone everything except
far few first for form found from further
gave get give given go goes going gone got
had has have having he her here hers herself him himself his how however
i if in indeed inside instead into is it its itself
just
keep kept know known
last later least left less let like likely little long
made make many may maybe me might more most much must my myself
near need never new next no none nor not nothing now
of off often on once one only onto or other others our ours out outside over own
part perhaps put
quite
rather really right
said same saw say says see seem seemed seen several shall she should side since
so some someone something soon still such sure
take taken than that the their theirs them themselves then there these they
thing things think this those though through thus to too took toward towards
turn turned two
under until up upon us use used using usually
very
was way we well went were what when where whether which while who whom whose why
will with within without would
yes yet you your yours yourself
""".split())

MIN_TERM_LENGTH = 2


def ensure(connection) -> None:
    """Creates the index if it is not there. Idempotent, and SQLite-only.

    Called from `migrations.bootstrap` on both paths — the fresh database that
    `create_all` just built, and the existing one the migration list is walking
    — because neither path can reach a virtual table on its own.
    """
    if connection.dialect.name != "sqlite":
        return
    connection.execute(sql(DDL))


def index_line(heading_path: str, text_: str) -> str:
    """What actually goes into the index for one passage."""
    return f"{heading_path}\n{text_}" if heading_path else text_


def add(db: Session, chunk_id: int, heading_path: str, text_: str) -> None:
    """Indexes one passage. The caller supplies the chunk's id as the rowid.

    `OR REPLACE`, and the reason is a defect M9 found rather than a defensive
    habit. The rowid is a chunk's primary key, so a row already sitting at it is
    by definition stale: the chunk that owned it does not exist, or is being
    rewritten by the reindex that called this. Either way the new passage is the
    truth and the old row is not.

    Without it, an orphaned index row makes an ordinary import fail. SQLite
    reuses primary keys once the highest row is gone, so the next campaign to
    import a source is handed rowid 1 again, collides with an orphan, and gets a
    500 from `INSERT` — and `clear_index` cannot clear the orphan, because it
    finds index rows *through* the chunks, and there are none. That made Reindex,
    which is the documented repair, unable to repair this. `REPLACE` closes it
    from both ends: a leaked row is overwritten the moment the id comes round
    again, so an existing database repairs itself rather than needing a
    migration, and Reindex is the repair it is described as.

    The leak itself is closed separately, in `importer.clear_campaign_index`.
    """
    db.execute(
        sql(f"INSERT OR REPLACE INTO {TABLE} (rowid, text) VALUES (:id, :text)"),
        {"id": chunk_id, "text": index_line(heading_path, text_)},
    )


def remove_adventure(db: Session, adventure_id: int) -> int:
    """Drops every index row belonging to one campaign. Returns how many.

    Scoped through the chunks, which is the only place the campaign is
    recorded — the index deliberately holds no copy of it
    (see "The table" above). So this has to run **before** the chunk rows go,
    which is what `importer.clear_campaign_index` is for.
    """
    result = db.execute(
        sql(
            f"""
            DELETE FROM {TABLE} WHERE rowid IN (
                SELECT id FROM knowledge_chunks WHERE adventure_id = :adventure_id
            )
            """
        ),
        {"adventure_id": adventure_id},
    )
    return result.rowcount or 0


def remove_chunks(db: Session, chunk_ids: list[int]) -> None:
    """Drops passages from the index by id.

    Called before the rows themselves go, because a chunk id read back after
    the row is deleted is a chunk id nobody has. SQLite has no `IN` binding for
    a list, so the ids are formatted into the statement — they are integers
    this process just read out of its own primary-key column, never anything a
    caller supplied.
    """
    if not chunk_ids:
        return
    ids = ",".join(str(int(chunk_id)) for chunk_id in chunk_ids)
    db.execute(sql(f"DELETE FROM {TABLE} WHERE rowid IN ({ids})"))


def terms(text_: str) -> list[str]:
    """The searchable words in a piece of query text, in order, deduplicated.

    Order is kept because the caller weights the query by what it put first, and
    because a deterministic query is one a maintainer can reproduce.
    """
    seen: set[str] = set()
    out: list[str] = []
    for raw in _TERM_SPLIT.split(text_ or ""):
        word = raw.strip("'-").lower()
        if len(word) < MIN_TERM_LENGTH or word in _STOP or word in seen:
            continue
        seen.add(word)
        out.append(word)
    return out


def match_expression(words: list[str]) -> str:
    """An FTS5 MATCH expression that finds any of `words`.

    Each word becomes a quoted phrase, so nothing in it can be read as an
    operator, and the phrases are joined with OR because a knowledge query is a
    bag of scene terms rather than a requirement that all of them appear.
    """
    quoted = [f'"{word.replace(chr(34), chr(34) * 2)}"' for word in words]
    return " OR ".join(quoted)


def search(
    db: Session,
    adventure_id: int,
    words: list[str],
    limit: int,
) -> list[tuple[int, float]]:
    """The best-matching enabled passages in one campaign, as (chunk_id, score).

    The score is a positive relevance, larger being better. FTS5's `bm25()`
    returns a *negative* number whose magnitude grows with the match, which is
    the opposite convention to everything else in this subsystem, so it is
    negated here — once, at the boundary — rather than left for each caller to
    remember.

    Three filters are applied in SQL, before any row reaches Python:

    * `adventure_id`, which is the cross-campaign isolation rule
      (`IMPORTED-KNOWLEDGE-DESIGN.md` §66). It is not a convenience and it is
      not the frontend's job.
    * `enabled`, so a disabled source cannot win a slot (§48).
    * `index_state = 'ready'`, so a source whose import failed halfway cannot
      retrieve out of a half-built index.

    `limit` bounds what comes back before the Python-side reranking runs, which
    is the rule `TECHNICAL-DESIGN.md` §13.1 records: candidates are capped in
    the database, not loaded and filtered afterwards.
    """
    if not words:
        return []
    rows = db.execute(
        sql(
            f"""
            SELECT c.id AS chunk_id, bm25({TABLE}) AS score
            FROM {TABLE} f
            JOIN knowledge_chunks c ON c.id = f.rowid
            JOIN knowledge_sources s ON s.id = c.source_id
            WHERE {TABLE} MATCH :query
              AND s.adventure_id = :adventure_id
              AND s.enabled = 1
              AND s.index_state = 'ready'
            ORDER BY score
            LIMIT :limit
            """
        ),
        {
            "query": match_expression(words),
            "adventure_id": adventure_id,
            "limit": limit,
        },
    ).all()
    return [(int(row.chunk_id), -float(row.score)) for row in rows]


#: How many query terms the evidence query asks about. The ranking query above
#: may carry more; this one becomes a subquery per term, so it is capped to keep
#: a single statement a sensible size. The terms are taken in query order, which
#: puts the current scene's own words first.
EVIDENCE_TERMS = 24


def term_evidence(
    db: Session,
    adventure_id: int,
    words: list[str],
    limit: int,
) -> dict[int, frozenset[int]]:
    """Which of `words` each candidate passage actually matched.

    Returns `{chunk_id: frozenset(index into words)}`.

    Admission needs to know *how much* matched, not merely that something did.
    FTS5's `bm25()` folds term count and rarity into one opaque number with no
    fixed range, and FTS5 has no `matchinfo()`, so the honest way to get a
    per-term answer is to ask per term — which is done here as a single
    statement with one subquery per term, rather than one round trip per term.
    Stemming is applied by FTS itself, so `resurrected` in the query matches
    `resurrection` in the passage exactly as the ranking query does; doing this
    in Python would need a second, divergent stemmer.

    The whole union is scoped once, at the join, so a term can never surface a
    passage from another campaign, a disabled source, or a source whose index is
    not ready.
    """
    words = words[:EVIDENCE_TERMS]
    if not words:
        return {}
    union = " UNION ALL ".join(
        f"SELECT {i} AS term, rowid AS chunk_id FROM {TABLE} "
        f"WHERE {TABLE} MATCH :w{i}"
        for i in range(len(words))
    )
    params = {f"w{i}": match_expression([word]) for i, word in enumerate(words)}
    params.update({"adventure_id": adventure_id, "limit": limit})
    rows = db.execute(
        sql(
            f"""
            SELECT t.term AS term, t.chunk_id AS chunk_id
            FROM ({union}) t
            JOIN knowledge_chunks c ON c.id = t.chunk_id
            JOIN knowledge_sources s ON s.id = c.source_id
            WHERE s.adventure_id = :adventure_id
              AND s.enabled = 1
              AND s.index_state = 'ready'
            LIMIT :limit
            """
        ),
        params,
    ).all()
    evidence: dict[int, set[int]] = {}
    for row in rows:
        evidence.setdefault(int(row.chunk_id), set()).add(int(row.term))
    return {chunk_id: frozenset(terms) for chunk_id, terms in evidence.items()}