Corpus Retrieval Contract

The retrieval interface (issue #66) is the one read path over the Product Expertise Corpus (#47). Downstream consumers — the Recommendation & Right-Sizing engine (#4) and the AI/LLM Narrative Feed (#10) — pull relevant, citation-ready passages through this interface rather than each reinventing retrieval. It lives in src/lib/server/corpus-retrieval.ts; the data it reads is the corpus store (#63, see corpus-data-model.md), populated by ingestion (#64, see corpus-ingestion.md).

This document is the contract. The inputs, result shape, and exclusion semantics below are what #4 and #10 may depend on. Anything not documented here (scoring weights, term-derivation details) is an implementation detail that may be tuned without notice — relative ranking may change; the shape and the guarantees may not.

The two query arms

retrieveBySystem(db, systemId, opts?)            → Promise<RetrievedPassage[]>
retrieveBySelectionContext(db, context, opts?)   → Promise<RetrievedPassage[]>

Arm	Input	Semantics
By system	a systems.id (e.g. sl45)	Passages relevant to that system, matched on its name/id and faceting vocabulary (operation type, material, interior/exterior, use cases)
By selection context	SelectionContext — { optionIds: string[] }, the same type the recommendation engine ($lib/server/recommend) consumes	Passages relevant to what the buyer chose, matched on option labels/values plus the factor names and the attribute names each option influences (factor_option_attributes)

SelectionContext is shared with #4 by construction: the recommendation engine's input passes straight through — one type, no adaptation layer.

Options (both arms)

Option	Default	Meaning
limit	8	Maximum passages returned
contentType	—	Narrow to one corpus content type: pdp | blog | tech-doc | marketing

Miss behavior

Retrieval never throws for a miss — every miss degrades to []:

• unknown systemId → []
• empty optionIds or all option ids unknown → []
• no passage scores above zero → []

A consumer rendering "no supporting content" is the correct degradation. A retired system still retrieves — its documentation remains valid historical material; only corpus staleness is excluded (below).

Result shape

Results are passage-level, ranked most-relevant first, each carrying full provenance so every quoted passage is citable:

type RetrievedPassage = {
	heading: string | null; // section heading the passage sits under
	seq: number; // passage order within its item's extracted text
	text: string; // the passage text
	score: number; // relevance (> 0); comparable within ONE query only
	provenance: PassageProvenance;
};

type PassageProvenance = {
	itemId: string; // corpus item id
	title: string; // document title, as cited
	contentType: CorpusContentType;
	sourceType: string; // 'url' | 'upload'
	sourceUrl: string | null; // public source URL (null for uploads)
	sourceUrlOk: boolean | null; // liveness (#1008): true = verified live, false = dead, null = unchecked
	sourceFilename: string | null; // uploaded filename (null for URL items)
	resourceTypes: string[] | null; // retained field_resource_type term names (#1480); null = none / pre-retention row
	version: string; // version label of the capture
	fetchedAt: string; // when the capture was fetched (ISO 8601)
};

Citation fields: a consumer citing a passage should render the document (title), the source (sourceUrl or sourceFilename), and the capture version (version) — the source + version pair is the corpus provenance contract (#63). itemId keys back to the corpus store for provenance walks (traceSupersession) or the full item (getCorpusItem).

No dead links (#1008): render sourceUrl as a link only when sourceUrlOk === true — dead (false) and unchecked (null) URLs degrade to the title with no anchor, fail-closed. The buyer-facing shapers (recommendation-evidence.ts, handoff.ts) already apply this gate; any new consumer must do the same.

Ordering is deterministic: equal scores tie-break by (itemId, seq), so the same corpus and query always produce the same result list.

CAD exclusion from the warm hand-off (#1480): CAD/Revit-type resources (Revit .rvt/.rfa, AutoCAD .dwg/.dxf, SketchUp, IFC) belong to the detailed-design phase, not the recommendation stage — so the hand-off shaper (resourceDocsFromPassages in handoff.ts) drops them from resources.documents. Because the filter sits on that single seam, every consumer of the payload inherits it: the results resource card, the configurator / contact-sales hand-off (#456), and the NanaSage → advisor escalation (#768). Classification is isCadResource (corpus.ts): taxonomy-first — an item's retained resourceTypes terms are authoritative when present (a CAD-vocabulary term ⇒ CAD; retained non-CAD terms ⇒ kept, even with a CAD-ish title) — with a title/filename-extension heuristic fallback covering rows ingested before term retention (a re-dump backfills them; audit 2026-07-15: 156 of 764 active tech-doc items classify as CAD, validated in cad-audit.spec.ts). This is a presentation filter only: CAD nodes still ingest, their link-stub passages remain non-scorable (#1414), retrieval scoring is unchanged, and the canonical Resources ↗ deep-link still reaches every CAD download on nanawall.com.

Exclusion guarantee

Retired and superseded items never appear in results. An item with retired_at set or superseded_by pointing at a replacement is stale; both arms drop its passages via the shared pure filter (retrievePassages, using the corpus store's isStale predicate) and the passage loader additionally pre-filters in SQL. Re-ingestion that supersedes an old capture (#64) therefore swaps retrieval over to the new version atomically with the supersession write — consumers do nothing.

Facet-tag consumption (#454)

The by-selection-context arm also consumes accepted content→facet tags, so ingested content that lexical matching alone would miss can still surface. This is the read side of the content enrichment engine: an admin accepts a tag classifying a corpus item under a shared-vocabulary facet (an attribute or factor), and that acceptance makes the item retrievable by that facet.

How it feeds the arm:

1. The selection context is resolved to the facets it implies — the factors of the selected options plus the attributes those options influence (factor_option_attributes). This is the same attribute/factor vocabulary the accepted tags are expressed in.
2. Every accepted tag whose facet is in that set contributes a per-item boost (facetTagBoosts); an item earns the boost once per matching tag.
3. The boost is added to each of the item's passage scores during ranking, so a passage can clear the "score > 0" bar — and thus surface — on the strength of a human-curated tag even with no lexical term hit.

Contractual guarantees:

• Only accepted tags are consumed — proposed and rejected tags never affect retrieval.
• Facet-tag consumption is additive: it can only raise an item's score, never lower or exclude it. The exclusion guarantee is unaffected — a stale item stays excluded regardless of its tags.
• The by-system arm is unchanged; facet-tag consumption is specific to the guided-selection arm.

The boost magnitude and the exact score contribution are implementation details (tunable without notice); the guarantee that an accepted, context-matching tag can only help an item surface is the contract.

Matching model (informative, not contractual)

Matching is hybrid — deterministic lexical scoring fused with semantic vector similarity (#1187, decision #1195). This is an implementation detail: the two arms, the RetrievedPassage[] shape, provenance completeness, and the retired/superseded exclusion are the contract; how passages are ranked within a query is not, and may be tuned without notice.

• Lexical arm (always on, the fallback): query terms derive from the domain entities themselves (system rows; factor options and their attribute mappings), are matched token-exactly ("SL45" never matches "SL4500"), and hits in a passage's heading or document title outweigh body hits. Body hits are BM25-style tf-saturated and passage-length normalized (#1408) — repetition asymptotes rather than accumulating, and the longer the passage the more hits it needs to reach the same score — so a giant PDF-dump passage (#1367 ingestion) can no longer bury concise, directly-relevant passages on raw term volume. With the vector arm off this is the entire ranking.
• Vector arm (opt-in): corpus passages are embedded (@cf/baai/bge-base-en-v1.5, 768-dim) into the nanaselect-corpus-text Vectorize index (#1186); a query embeds its terms and pulls the nearest passages. See rag-intelligence-bindings.md.
• Fusion: the two arms are combined by Reciprocal Rank Fusion (RRF) with tunable weights — score = w_lex·1/(k+lexRank) + w_vec·1/(k+vecRank). RRF fuses by rank, not by raw score, because lexical counts and cosine live on incomparable scales; this keeps exact matches anchored (they score on both arms) while letting a relevant-but-lexically-missed passage surface.

Degradation (never fails): the vector arm is gated by a default-OFF flag (CORPUS_HYBRID_RETRIEVAL) and resolved from ambient request bindings. Flag off, no request context, a missing binding, or an inference error ⇒ the retriever silently falls back to pure lexical scoring. Corpus staleness (retired/superseded) is enforced on the authoritative D1 load before fusion, so a stale Vectorize vector can never surface a passage the contract excludes.

Rollout: OFF in production until the hybrid beats the lexical baseline on its eval bar (#1192, #1193). The eval harness npm run corpus:eval measures recall lift on a documented paraphrase eval set (data/eval/corpus-retrieval-eval.json) whose relevant passages are lexically divergent from the query. On that set, at the default retrieval limit (8), lexical retrieves 0/6 of the target passages while hybrid retrieves 5/6 — a GO on the eval bar (recall@8 0.83 ≥ floor 0.8). Re-baselined after the #1408 length normalization: saturating the lexical arm stopped giant dump passages flooding the RRF rank list, which lifted the hybrid from its earlier 3/6 to 5/6 (the flag flip itself remains a human call, #1193).

The pure functions (tokenize, systemQueryTerms, contextQueryTerms, scorePassage, rankPassages, retrievePassages, blendVectorScores) are exported and unit-tested (corpus-retrieval.spec.ts).