Refresh the Corpus from nanawall.com Content

What/why: Content & media — what this screen manages and why it exists.

Purpose: feed public nanawall.com content — product detail pages, blog posts, technical documentation — into the Product Expertise Corpus (#47), and keep it current as the site changes.

When to use: populating a fresh dev database, adding newly published site content, or refreshing captures after site pages change.

Since #457 a scheduled liveness pass also runs weekly (Mondays 04:23 UTC): the content-refresh worker re-checks source-URL liveness so the render layer's fail-closed link gate (#1008) works from fresh data — see the freshness governance reference and the Coverage & freshness panel on /admin/corpus. (Until #1099 the worker also re-ran the JSON:API content sources on cadence; those run-paths are retired — corpus content refreshes via the paths below.)

The app runtime never scrapes the live site (#65). Content is supplied out of band as JSON files under data/corpus/, then pushed through the ingestion pipeline (#64) by an operator-run script. Refreshing the corpus = refreshing the files + re-running the script; ingestion converges idempotently, so the whole directory is safe to re-run any time.

For one-off uploads, replacements, and retiring documents, data editors can use the /admin/corpus screen instead — no local tooling needed. It feeds the same pipeline with the same document contract; see managing-corpus.md.

Bulk nanawall.com content no longer arrives through files here at all — the DB-dump pipeline (npm run drupal:load, #1369 single path) ingests it straight into corpus_items; see corpus ingestion. This file pipeline remains for hand-provided documents (uploads). The megamenu product-family pages (#1961) ride the same standard dump refresh — extract → load → corpus:embed — with a load-time coverage gate that fails loudly if any enumerated family URL is missing from the corpus.

Source file format

One JSON file per document, under a directory named for its content type:

data/corpus/
├── pdp/        product detail pages        → contentType "pdp"
├── blog/       blog posts                  → contentType "blog"
└── tech-doc/   technical documentation     → contentType "tech-doc"

Each file is a corpus ingestion document:

{
	"sourceType": "url",
	"sourceUrl": "https://www.nanawall.com/products/sl45", // the page captured
	"title": "SL45 — Aluminum Framed Folding Glass Door",
	"contentType": "pdp", // matches the directory
	"version": "2026-07-02", // source-declared date (blog publish date)
	// or the capture date when the page declares none
	"sourceDate": null, // page publish/modified date when known (ISO 8601)
	"passages": [
		// the page's content sections, in order
		{ "heading": "Overview", "text": "…" }
	]
}

Mapping conventions for nanawall.com content:

Site content	contentType	version	sourceDate
Product detail page (/products/…)	pdp	capture date	null (pages are undated)
Blog post (/blog/…)	blog	publish date	publish date
Technical/performance page (/performance/…, /support/…)	tech-doc	capture date	null unless the page shows one

Passages should follow the page's own section structure — one passage per heading, text condensed to the substantive content (drop navigation, CTAs, boilerplate). The item id is derived from the URL path automatically (/products/sl45 → products-sl45); don't encode ids in filenames — the filename is only a human label.

Refresh workflow

1. Capture the page content out of band (browser, curl, or any fetch tool — this step is manual/operator tooling by design) and update the document's passages.

2. Stamp the version: the new publish date (blog) or today's date (undated pages). Update sourceDate if the page declares a date.

3. Re-run ingestion:

   npm run corpus:ingest                 # everything under data/corpus/
   npm run corpus:ingest -- <file>       # one document

4. Read the summary. Each file reports created, updated, unchanged, or REJECTED (with the validation problems). An unchanged file is a no-op; a refreshed page updates the existing item in place — same id, one row, never a duplicate. Rejected files fail the run's exit code but don't block other files.

By default the runner targets the local dev database and R2 (wrangler platform proxy). To populate the production corpus, use --remote — see below.

Populating the production corpus (--remote)

The ingest + embed operator scripts accept --remote (#1224), which points the D1 (DB) and corpus R2 (CORPUS) bindings at the production account instead of the local persisted store. (Vectorize and Workers-AI are already remote in every run — they have no local emulation — so --remote only moves the D1/R2 half.)

Remote writes are guarded so a prod mutation can never be accidental:

• the script prints the exact target (D1 database name + id, the CORPUS bucket, the Vectorize indices) before writing;
• it requires an explicit confirm — pass --yes, or answer the interactive prompt. In a non-interactive shell (CI, a pipe) it refuses unless --yes is given;
• --dry-run reports what would be written (counts + target) and mutates nothing — always safe to run first.

Runs stay idempotent and budget-bounded exactly as locally: the same per-run cap, monthly spend ledger, and retry/degrade governance (#1191) apply, now measured against production spend. An interrupted run resumes cleanly — a re-run skips everything already ingested/embedded (unchanged content is a no-op).

# 1. Preview against production (no writes):
npm run corpus:ingest -- --remote --dry-run
npm run corpus:embed  -- --remote --dry-run
npm run media:embed   -- --remote --dry-run

# 2. Apply to production (prints the target, then writes after --yes):
npm run corpus:ingest -- --remote --yes            # documents → prod D1 + R2 CORPUS
npm run corpus:ingest -- --remote --yes --embed    # ingest, then embed changed passages
npm run corpus:embed  -- --remote --yes            # backfill passage vectors → prod
npm run media:embed   -- --remote --yes            # backfill media vectors → prod

The media analogue (npm run media:embed) works identically — same --remote / --yes / --dry-run flags, same guard and budget governance. It embeds referenced image assets into the nanaselect-media-sim index and writes the media_embeddings provenance row; --remote targets the production D1 for that row (the vector index is already remote). The media R2 bucket (MEDIA) stays local by design and is never touched by these scripts.

Verifying a refresh

npx wrangler d1 execute DB --local --command \
  "SELECT id, content_type, version, fetched_at FROM corpus_items WHERE id NOT LIKE 'seed-%'"

# Same query against production (swap --local for --remote):
npx wrangler d1 execute DB --remote --command \
  "SELECT id, content_type, version, fetched_at FROM corpus_items WHERE id NOT LIKE 'seed-%'"

Provenance stays intact through refreshes: version/fetched_at move to the new capture, source_url and the stable id never change, and content_hash reflects the current passages (see the corpus data model).