Corpus Data Model

The corpus store (issue #63) is the keystone of the Product Expertise Corpus (#47): the persisted body of NanaWall product knowledge — product detail pages, blog posts, technical documentation, marketing collateral — that the recommendation, narrative, and advisor layers (#4, #10) retrieve from and cite. Structured data (systems × attributes) says which system fits; the corpus lets the engine explain why, grounded in the real product literature.

Storage splits by size and shape:

Where	What
D1	Item metadata, provenance, lifecycle, and the extracted text passages used for retrieval
R2	Large source binaries (PDFs, spec sheets, collateral), referenced by key from the D1 row

The R2 bucket is bound as CORPUS (nanaselect-corpus) in wrangler.jsonc. Local dev uses an emulated bucket automatically. Pre-deploy precondition: the remote bucket must exist before the next wrangler deploy — create it once with wrangler r2 bucket create nanaselect-corpus.

Schema — corpus_items

Defined in src/lib/server/db/schema.ts (migration drizzle/0015_square_black_queen.sql). One row per document capture.

Column	Type	Meaning
id	text PK	Stable kebab-case id (URL-derived or upload slug); derivation is owned by ingestion (#64)
title	text	Document title, shown to admins and cited by the narrative layer
content_type	text	pdp | blog | tech-doc | marketing — the corpus content taxonomy (#47)
source_type	text	url (public nanawall.com content) | upload (admin-contributed material)
source_url	text	Public source URL; NULL for uploads
source_url_ok	integer	Liveness of source_url (#1008): 1 = verified live (2xx), 0 = dead, NULL = not yet checked
source_url_checked_at	text	When source_url_ok was last verified (ISO 8601); NULL = never checked
source_filename	text	Original uploaded filename; NULL for URL-sourced items
version	text	Version label of this capture: source-declared version, source-modified date, or capture date
source_date	text	When the source itself was published/last modified (ISO 8601), if known
fetched_at	text	When this capture was fetched/uploaded (ISO 8601)
content_hash	text	SHA-256 of the extracted text — the idempotency key for version-aware re-ingestion (#64)
r2_key	text	Key of the source binary in the CORPUS bucket; NULL = no stored binary (e.g. an HTML page)
mime_type	text	MIME type of the stored binary; NULL when no binary
size_bytes	integer	Size of the stored binary; NULL when no binary
superseded_by	text FK	Self-reference to the newer capture that replaces this one; NULL = current
retired_at	text	Soft-delete marker (ISO 8601); NULL = active
created_at	text	Row creation (ISO 8601)
updated_at	text	Last row change (ISO 8601)

Schema — corpus_passages

The retrieval unit: ordered segments of an item's extracted text. Retrieval (#66) and the narrative layer quote passages, not whole documents, and heading lets a citation say where in the document the passage came from.

Column	Type	Meaning
id	integer PK auto	Row id
item_id	text FK	Owning corpus_items.id (cascade on hard delete)
seq	integer	Passage order within the item, 0 first — unique per item
heading	text	Section heading/breadcrumb the passage sits under, if any
text	text	The extracted passage text

Passages carry no lifecycle of their own: an item's supersede/retire state governs all of its passages.

Provenance model

Every item answers three provenance questions from its own row:

1. Where did this come from? source_type + source_url / source_filename.
2. Which capture of the source is it? version, source_date, fetched_at, and content_hash (the machine-comparable identity of the extracted text).
3. Is it still current? superseded_by chains an old capture to its replacement; retired_at soft-deletes stale material outright.

Lifecycle states

State	Representation	Retrieval visibility
Active	superseded_by and retired_at both NULL	Retrievable
Superseded	superseded_by set	Excluded; traceable via the supersession chain
Retired	retired_at set	Excluded; preserved for audit (never hard-deleted)

Soft-delete mirrors the knowledge-base convention (#18/#19/#20): stale material is findable and retirable, never destroyed. Ingestion (#64) updates in place: a refreshed capture of the same source keeps its stable id — one row per source, never a duplicate (see corpus-ingestion.md). The superseded_by chain is for distinct captures registered explicitly — a relocated source, a curator-driven replacement — not for routine re-ingestion.

The lifecycle is admin-managed: data editors retire and restore items (and upload replacements) from the /admin/corpus screen, audited under the corpus entity — see managing-corpus.md.

Source-URL liveness — the no-dead-links guarantee (#1008)

No link surfaced on the results view may 404. Liveness is enforced at the authoritative point — stored on the item and filtered at render — never probed per-request:

• source_url_ok is stamped by a HEAD (GET fallback) probe: inline when a url-sourced item is ingested/refreshed, and on demand via the Recheck links action on /admin/corpus. The pass covers every URL-bearing source type — url, drupal-view, drupal-dump (#1422) — and is bounded (default 300 items per run, never-checked first, then stalest check first) so one run stays inside the Workers subrequest budget; repeated runs sweep the whole corpus, and the action reports how many items the current sweep hasn't reached yet.
• The render-time gate is fail-closed: only source_url_ok = 1 renders as a link. Dead (0) and unchecked (NULL) both degrade to the document title with no anchor; a section left empty collapses per the existing empty-state rules. The gate is the shared citation-link rule ($lib/citation-links.ts, #1422) used by the evidence/citation shapers (recommendation-evidence.ts, nanasage/citations.ts) plus handoff.ts.
• Derived deep-links ($lib/deeplink) follow the same rule: the PDP link renders only when a pdp_path was captured; Configure & Price / Resources are id-templated nanawall.com routes and are subject to the same gate if ever found dead. No raw slug is ever rendered as a link.

Seed data

seed/sql/070-corpus.sql covers all lifecycle states — an active URL-sourced PDP, an active uploaded tech-doc (with a placeholder r2_key), a superseded blog capture chained to its replacement, and a retired marketing brochure — so retrieval filtering and provenance queries are exercisable against a fresh dev database. It also seeds one active url-sourced blog with source_url_ok = 0 (the FSW75 dead-resource case) so the title-only fail-closed render path is exercisable in dev.

Consumers

• #64 — ingestion pipeline: writes items/passages, drives supersession via content_hash.
• #66 — retrieval interface: reads active items' passages by system and selection context.
• #4 / #10 — recommendation & narrative: cite passages with provenance.