Corpus Ingestion Contract

The ingestion pipeline (issue #64, src/lib/server/corpus-ingest.ts) is the write path that populates the corpus store (#63). Content arrives as provided JSON payloads or downloaded files — the app runtime never scrapes the live site, mirroring the CSV seed stance (#15). Ingestion is idempotent and version-aware: re-ingesting a refreshed document updates the existing item in place; re-running an unchanged one is a no-op.

Four writers feed it, all holding this exact provided-document contract: the operator-run corpus:ingest CLI (corpus-refresh.md), the /admin/corpus upload form (managing-corpus.md) which passes the submitted JSON straight through (an attached source file becomes binary), the content-source registry's text arm (#343, drupal-content-registry.md), and — new in #415 — the registry's view arm, which ingests the nanawall.com iPad-app JSON:API views (app_resources, app_brochures, app_project_case_study) as UUID-keyed drupal-view documents.

Fetch-stance seam (#343, #415)

The registry lives in the sync layer and does fetch nanawall.com live, which would appear to conflict with this module's "never scrape" stance. It does not: the seam is resolved so that ingestion stays provided-only. The registry is the fetch arm — it fetches a Drupal node, normalizes it into exactly this document format, and hands the result to ingestCorpusDocument as a provided document. corpus-ingest never gains a fetch arm; the "never scrape" contract here is preserved, not inverted. Since #1369 the DB-dump pipeline is the SINGLE bulk ingestion path — the live JSON:API fetch arms below are retired (described for history; their declarative contracts remain in force):

• Text arm (#343) — a JSON:API node collection (jsonapi/node/{type}) paged by an nid cursor, one body field per node (src/lib/server/drupal/text-to-corpus.ts).
• View arm (#415) — a JSON:API view (jsonapi/views/{view}/default) paged by the view's own ?page=N pager, carrying multiple ordered prose fields (summary + body + testimonial) and an included PDF; normalized by src/lib/server/drupal/view-to-corpus.ts.
• Dump arm (#1365, #1366) — no fetch at all: npm run drupal:load reads text AND view content straight from the neutral DB-dump extract, driven by the same TEXT_SOURCES / VIEW_SOURCES field contracts as the fetch arms (bundle from the endpoint's last segment, field tables from Drupal's node__<field> naming rule), normalized by src/lib/server/drupal-dump/corpus-arm.ts. Documents ingest with sourceType: 'drupal-view' so the node UUID is the id basis — the dump and view arms converge on the same row for the same node. Liveness probes are skipped during the load (the pipeline is offline by contract); the scheduled #1008 liveness pass covers dump-ingested items afterwards. After ingest the load runs an identity-convergence pass (supersedeUrlKeyedTwins): a URL-keyed row the text arm wrote for the same source_url is marked superseded by the UUID-keyed row, so retrieval never serves the same content twice (full fetch-arm retirement is #1369). Long documents are written as multiple ≤20-row passage INSERTs inside the one atomic batch, respecting D1's ~100 bound-parameter cap (#1380).

View membership from raw tables (#1366)

A raw dump has no Drupal "views" — each iPad-app view's node selection is reproduced from its exported view config (nanawalld8 config/default/views.view.<view>.yml), recorded in VIEW_MEMBERSHIP (src/lib/server/drupal-dump/corpus-arm.ts):

View	Membership from dump tables
app_resources	published resource nodes (the config's product/resource-type filters are exposed; the JSON:API wire fetch does not apply exposed defaults)
app_brochures	published resource + field_resource_type ∈ {eBooks, Marketing Brochure, Product Brochures & eBooks} + field_product term carries field_app = 1
app_project_case_study	published case_study nodes

Spot-check (2026-07-14): brochures matched the live view exactly (16/16). The two large views under-return on the live wire — their unstable sorts duplicate rows across page boundaries (19 dup rows in the first 100) and crowd distinct nodes out (609/779 resources, 203/228 case studies reachable) — so the dump arm, which reproduces the config rather than the pager artifact, ingests the complete membership (it recovered the 25 pager-lost case studies).

Because app_brochures selects a subset of app_resources, view sources are processed most-specific first (VIEW_PRECEDENCE): a UUID claimed by brochures (→ marketing) is skipped by the resources sweep (→ tech-doc) — one node, one row, one content type. A content-type re-route counts as an update in planIngest even when the text is identical, so a reclassified node converges instead of being skipped. Case studies emit passages in the declared order — Summary → body → Testimonial — with section headings.

PDP arm — taxonomy-driven (#1233)

PDP content is not a node collection: a products taxonomy term names the system and references its page via field_product_node → a published product_detail node — whose prose lives on a paragraph tree (tabs → segments → grids/accordions → text blocks), not in node__body. The dump arm reproduces this without any live fetch (extractPdpDocs, src/lib/server/drupal-dump/corpus-arm.ts; mechanism decision #1409 — the JSON:API taxonomy fetch the original AC named was retired with the fetch arm):

Step	Dump tables
Membership	taxonomy_term_field_data (vid = products) → taxonomy_term__field_product_node → published product_detail node
Passages	paragraphs_item_field_data parent chain walked from the node (DFS, cycle-safe, depth ≤ 8); text from paragraph__field_text, headings from paragraph__field_heading_text; a section_heading paragraph titles everything after it at its level; the term's own description leads as an Overview passage
Provenance	contentType: 'pdp', sourceType: 'drupal-dump', id off the node UUID, canonical URL from path_alias, version = node changed date

The extract whitelist carries paragraphs_item_field_data + the paragraph__field_ prefix for this arm (re-run npm run drupal:extract on an older extract that predates them). 2026-07-15 run: 22 systems, 23–43 passages each, zero warnings. After ingest, supersedeLegacyPdpRows converges the hand-dropped data/corpus/pdp/ samples (removed from the repo in the same change): a live non-dump pdp row is superseded by the dump row for the same URL slug — the samples used the legacy /products/<slug> path while the dump carries the current /glass-walls/…/<slug> alias, and for PDP rows the slug is the system id (#414), so exact-URL twin matching would miss them.

Family-page arm — enumerated aliases (#1961)

The nanawall.com megamenu product-family pages (/glass-walls, /folding, /sliding, /stacking, /frameless, /foldflat, /windows, /floor-supported-systems, /slimline-steel-effect, /hurricane-zone-doors, /bear-resistant-doors, /insect-screens, /pricing — 13 pages) carry the family-level "when to choose this" guidance rule authoring cites. Membership is an enumerated alias contract (FAMILY_PAGES, src/lib/server/drupal-dump/corpus-arm.ts), not a bundle sweep: the pages are basic_page nodes (a bundle that also carries legal/utility pages which must never ingest), except FoldFlat — a product_detail outside the products-taxonomy join, which therefore rides this arm. The interactive tool pages under the same menu (/glass-walls/compare, /glass-walls/finder) are deliberately excluded (no guidance prose).

Step	Dump tables
Membership	path_alias (enumerated alias → nid) → published node in node_field_data
Passages	same paragraph walk as the PDP arm (buildParagraphIndex / walkParagraphTree are shared)
Provenance	contentType: 'family', sourceType: 'drupal-dump', id off the node UUID, live nanawall.com URL from the alias

Coverage gate (admin tier, fail loud): after every npm run drupal:load, each enumerated family URL must have a live corpus item — a gap fails the load's exit code listing the missing URLs. The family arm runs last in claiming order, so a family node a more specific arm ever comes to own yields to that arm (the gate checks URLs, not arm ownership). Family pages ride the standard dump refresh: extract → load → corpus:embed — no separate refresh path.

Linked-file ingestion (#1367)

Corpus-member nodes carry their real substance as attached files (node__field_files → file_managed), not prose. During drupal:load:

1. Resolve (pure, src/lib/server/drupal-dump/file-ingest.ts): each member node's file references join through file_managed (delta-ordered, unmanaged/deleted rows skipped, dangling targets warned — never fatal).
2. Fetch offline (load CLI only — the Worker never fetches): extractable files are downloaded from --files-base-url (default https://www.nanawall.com/sites/default/files) with an on-disk cache at <extract-dir>/.files-cache/, so re-runs never re-download.
3. Extract: PDF pages become ordered corpus passages appended to the owning node's document — the first carries a Linked document: <name> heading so retrieval citations name the file. The passages join the item's content hash, so a changed PDF triggers an update and an unchanged re-run is a no-op.
4. Store: fetched binaries land in the CORPUS R2 bucket at corpus/files/<fid>/<filename> — fid-keyed, so a file linked from many nodes is fetched and stored once — and the owning item's r2_key/mime_type/size_bytes point at its primary (first PDF) collateral via attachCorpusCollateral.
5. Record, don't fail: non-extractable types (CAD .dwg, .docx, octet-stream) and fetch failures become a Linked file: <name> passage carrying filename, type, size, and public URL — the load summary reports extracted / binary-only / fetch-failed / stored counts. Supported for text extraction today: PDF (via unpdf, a load-CLI-only devDependency — see decision #1389). A remote (--remote) load uploads the same binaries to the prod bucket through the R2 REST API (RemoteR2Bucket, same API token as remote D1 — it needs R2 edit access; a refused upload degrades to a recorded file, never a failed load).

Entry point

ingestCorpusDocument(db, bucket /* R2 | null */, payload) → { status, id }
// status: 'created' | 'updated' | 'unchanged'

Throws CorpusIngestError (with a problems: string[] list) before any write when the payload is malformed — there is no partial ingest.

Accepted document format

{
	"sourceType": "url", // 'url' | 'upload' | 'drupal-view'
	"sourceUrl": "https://www.nanawall.com/products/sl45", // required for 'url' and 'drupal-view'
	"sourceFilename": "guide.pdf", // required for 'upload'
	"sourceUuid": "a1b2c3d4-…", // required for 'drupal-view' — the node UUID, its id basis
	"title": "SL45 Product Detail Page",
	"contentType": "pdp", // 'pdp' | 'blog' | 'tech-doc' | 'marketing'
	"version": "2026-06-20", // source-declared version or date label
	"sourceDate": "2026-06-15T00:00:00.000Z", // optional: source publish/modified date
	"passages": [
		// ordered extracted text — the retrieval units
		{ "heading": "Overview", "text": "…" }
	],
	"binary": {
		// optional: source file to store in R2
		"data": "<ArrayBuffer|Uint8Array>",
		"mimeType": "application/pdf"
	}
}

Validation collects every problem (missing title, wrong taxonomy value, absent source arm, empty passages, malformed binary) and rejects the document whole. The provenance invariants are then re-checked at the store boundary (validateCorpusItemInput), so nothing malformed can slip between layers.

Identity: one source → one row

The stable item id is derived deterministically from the source:

Source	Basis	Example
url	slug of the URL path (hostname for a bare domain)	https://…/products/sl45 → products-sl45
upload	slug of the filename, extension dropped	Acoustical Performance Guide.pdf → acoustical-performance-guide
drupal-view	the node UUID (the id is the uuid)	a1b2c3d4-… → a1b2c3d4-…
drupal-dump	the node UUID — same basis as drupal-view	a1b2c3d4-… → a1b2c3d4-…

drupal-dump (#1368) is the offline dump pipeline's own source type: identical UUID identity, so a node converges on ONE row no matter which arm delivered it — the dump arm relabels a view-arm row in place (a source-type change counts as an update in planIngest, like a content-type re-route) — while the origin is honestly recorded. The freshness panel attributes dump-loaded corpus rows as their own Drupal DB dump — corpus row (alongside the media/attributes dump rows), and each drupal:load records corpus counts (documents written, PDFs extracted) in the bulk-load audit event surfaced on the admin Drupal-data screen. Provenance keys are stable across re-runs — an idempotent reload is a no-op after the one-time relabel.

Because the same source always maps to the same id, re-ingestion converges on a single row — there is nothing to deduplicate after the fact. A drupal-view item is keyed to the node UUID rather than its URL path (#415), so a Drupal alias change never orphans it into a duplicate; its canonical URL is still retained in sourceUrl for liveness + citation. A view node's linked PDF (field_files) is referenced as a final Linked document passage (filename + URL) so it is retrievable as document content.

Idempotency & versioning

The decision against the store's current row (planIngest, pure):

Store state for the id	Result
No row	created
Same content_hash and same version	unchanged — no writes at all
Anything differs	updated in place — same id, capture fields (title, version, dates, hash, R2 reference) refreshed, passage set replaced

content_hash is SHA-256 over the ordered passages (headings + text), so "did the content actually change?" is answered by comparison, independent of version labels — a re-labeled but identical document is still unchanged in content terms only if both the label and hash match; a changed label alone triggers an update so provenance stays truthful.

A routine refresh never creates a supersession chain. superseded_by (#63) is for distinct captures registered explicitly — e.g. a relocated source or a curator-driven replacement — not for re-ingesting the same source.

Atomicity & resumability

• Validation happens entirely before any write: malformed input writes nothing (no D1 row, no R2 object).
• The R2 binary (when present) is written first, under the deterministic key corpus/<id>/<filename>. If the subsequent D1 batch fails, re-running overwrites the same key — a partial run is repaired by the next run, never left as drift.
• All D1 writes for a document (item upsert + passage replace) land in one atomic db.batch — no reader ever sees a half-ingested item.
• Re-running any input sequence converges: same inputs → same store state (INSERT OR REPLACE-style semantics, the #15 seed convention).

Verification

Unit specs (corpus-ingest.spec.ts) cover the contract, id determinism, hash sensitivity, and plan semantics. The live double-ingest (create → unchanged → updated, single row throughout; R2 object present; malformed rejected with zero writes) is exercised against local D1/R2 via wrangler's platform proxy.

Freshness governance (#457)

Ingestion records provenance (fetched_at, source_url, content_hash); governance keeps that provenance from silently going stale and keeps the app from ever presenting itself as the canonical source.

Canonical deep-links

nanawall.com stays canonical. Every URL-sourced item stores its canonical source_url at capture, and every buyer-facing render deep-links it, fail-closed on liveness (#1008): recommendation-evidence.ts and handoff.ts only render source_url as a link when source_url_ok === true — dead and never-checked URLs fall back to title-only, so no surface serves a 404 or claims the content as its own. Uploads (no public URL) render title + version. Project rows (#983) go further: a project with no resolvable canonical URL is excluded entirely.

Refresh cadence (scheduled)

workers/content-refresh is a cron-only companion worker (session-purge pattern, same D1 + CORPUS bucket) that re-checks source-URL liveness every Monday 04:23 UTC and stamps content_refresh.last_run (an app_settings row), so the fail-closed link gate above always works from fresh data. The pass is idempotent — safe at any cadence. Until #1099 the worker also re-ran the JSON:API content-source registry on the same schedule; those sync run-paths are retired, and corpus content now refreshes via the manual paths (/admin/corpus tools, corpus:ingest CLI) while catalog content refreshes via the DB-dump ingest (#1094). Deploy with npm run deploy:refresh; test locally with npm run refresh:dev.

Coverage & staleness admin

/admin/corpus carries a Coverage & freshness panel (src/lib/server/admin/content-freshness.ts, media-coverage pattern): per content type it shows active-item counts, accepted-facet-tag coverage, canonical-link coverage, the oldest fetched_at, and how many items are past the staleness threshold. The threshold is data, not a deploy — an app_settings row (corpus.staleness_days, default 90) editable on the panel and audited on change. Items past it surface a warning so a stale corner is a visible, actionable state.