Runbook: Refresh the NanaSage Corpus from a Drupal Dump

Purpose: Update the Product Expertise Corpus (blogs, resources, brochures, case studies, PDPs, linked PDFs) from a fresh nanawall.com Drupal MySQL dump — local dev (nanaselect.iwpi.com) or production. Every step is idempotent: re-running any step with the same inputs is a no-op, and an interrupted run resumes cleanly on re-run. When to use: Site content changed and NanaSage answers should reflect it; populating a fresh dev DB; syncing prod after a local refresh; a previous run was interrupted (just re-run it). Prerequisites: A Drupal MySQL dump (.sql or .sql.gz), obtained out of band — never commit it to the repo (it holds PII until extraction strips it). npm install done. For --remote (prod): CLOUDFLARE_API_TOKEN with D1 + R2 edit access. Estimated time: ~5 min extract + load (local); embeds vary with changed-passage count (budget-capped, resumable).

The DB-dump pipeline is the single bulk ingestion path (#1369 — the old JSON:API feed is retired). The /admin/corpus screen is only for one-off uploads, retire/restore, staleness threshold, and link re-checks — bulk refresh is this CLI flow. All commands run from the repo root.

Steps

1. Extract the dump into a neutral extract

npm run drupal:extract -- path/to/nanawall.sql.gz --out extract --drupal-version 10.3

Offline and local: the dump is parsed as a text stream — no MySQL server, no network. Excluded tables (users, webform, commerce, sessions, config) never leave the stream; only whitelisted content tables reach extract/. Pass --dump-date <iso> if the dump lacks a -- Dump completed on footer.

Expected output:

extract complete in 12.4s → extract/
  source: nanawall.sql.gz (db nanawall, dumped 2026-07-14 21:10:17, Drupal 10.3)
  tables: 41 whitelisted, 58231 rows
  skipped: 96 non-whitelisted tables (names in manifest)

Decision point: If the run warns about missing dump date or Drupal version, re-run with --dump-date / --drupal-version — the freshness view reports the dump's date, so a provenance gap stays visible forever. If you're re-using an existing extract/ (check source.dumpDate in extract/manifest.json) and it's fresh enough, skip to step 2.

2. Load the extract (stores + corpus + linked PDFs)

npm run drupal:load -- extract                    # local dev D1 (default)

One command does all of it: catalog stores (projects, media, attribute values), corpus documents keyed by node UUID (blogs, resources, brochures, case studies, PDPs), and linked PDFs fetched from nanawall.com, extracted into passages, and stored in R2. Downloads cache in extract/.files-cache/ so re-runs never re-fetch. The content diff — not timestamps — decides writes, so re-running the same extract is a no-op.

Expected output: per-store create/update/unchanged counts, corpus document counts, and a linked-file summary (extracted / binary-only / fetch-failed / stored). A bulk-load event is recorded (visible on the admin Drupal-data screen).

Decision point: If the load aborts with a schema-drift report (missing table/column), the dump's Drupal schema moved — see Troubleshooting. If it reports per-item failures, fix nothing blindly: re-run once (idempotent resume), then investigate what still fails.

3. Embed new/changed passages

npm run corpus:embed                              # passage vectors
npm run media:embed                               # media vectors (only if media changed)

Vectorize and Workers AI are always remote (no local emulation) — these commands cost real money but are budget-capped per run with a monthly spend ledger, and resume cleanly: already-embedded content is skipped.

Expected output: embedded/skipped counts and budget consumption. A second run immediately after reports everything skipped.

4. Verify

npx wrangler d1 execute DB --local --command \
  "SELECT id, content_type, version, fetched_at FROM corpus_items WHERE id NOT LIKE 'seed-%' ORDER BY fetched_at DESC LIMIT 20"

Expected output: rows stamped with the new dump's date. Also eyeball /admin/corpus — the item list plus the Coverage & freshness panel (per-type counts, oldest fetched_at, staleness warnings).

Decision point: Local-only refresh? Done. Syncing production? Continue to step 5.

5. Production (--remote) — same commands, guarded

Always dry-run first; --dry-run mutates nothing:

npm run corpus:embed -- --remote --dry-run

Then apply. --remote moves only the D1/R2 write target to prod (nanaselect-db + prod CORPUS bucket); the plan and all idempotency guarantees are identical:

export CLOUDFLARE_API_TOKEN=…                     # D1 + R2 edit access
npm run drupal:load -- extract --remote           # prompts: re-type the DB name to confirm
npm run corpus:embed -- --remote --yes
npm run media:embed  -- --remote --yes

Each command echoes the exact target (DB name, id, account, bucket, indices) before writing and requires confirmation — interactive prompt, or --yes for non-interactive shells (which otherwise refuse). Verify with the step-4 query using --remote instead of --local.

6. Clean up the dump

Delete the raw .sql[.gz] file. The extract is safe to keep (whitelisted content only); the dump is not (PII).

Troubleshooting

Symptom	Likely Cause	Fix
Load aborts with per-store schema-drift report	Dump's Drupal schema changed vs MAPPING_DRUPAL_VERSION, or the extract predates a whitelist addition (e.g. paragraph__field_* for PDPs)	Re-run drupal:extract on the dump with current code; if it persists, the mapping (src/lib/server/drupal-dump/mapping.ts) needs updating — that's a code change, not an ops step
Loader refuses the extract (format version)	Extract produced by an older/newer extractor	Re-run drupal:extract from the same dump with current code
Extract warns "no dump date"	Dump lacks the -- Dump completed on footer	Re-run with --dump-date <iso> — don't ship a null provenance date
--remote run refuses to write	Non-interactive shell without --yes, or missing/underscoped CLOUDFLARE_API_TOKEN	Pass --yes explicitly; token needs D1 edit (+ R2 edit for PDF upload)
Embed run stops early	Per-run budget cap hit (#1191)	Expected behavior — re-run later; it resumes where it stopped
PDF fetch failures in load summary	nanawall.com file URL dead or moved	Non-fatal by design — the file is recorded as a Linked file: passage; re-run after checking --files-base-url
Same document appears twice in retrieval	Should not happen — UUID identity converges arms onto one row	Check superseded_by on the twin rows; file a bug if both are active
Item shows in DB but dead link in results	source_url_ok false/unset (fail-closed link gate #1008)	Run Recheck links on /admin/corpus, or wait for the Monday 04:23 UTC scheduled pass

Escalation

If this runbook doesn't resolve the issue:

1. Check the deeper references: corpus ingestion contract, dump extraction pipeline, corpus refresh how-to, extract how-to.
2. Search the repo's GitHub issues for the pipeline family: #1094 (capability), #1095 (extract), #1097 (load), #1336 (remote), #1365–#1369 (corpus arms), #1233 (PDPs), #1367 (linked PDFs).
3. File a bug with gh issue create --label bug, attaching the run summary output and extract/manifest.json source block (never the dump itself).