Extract a Neutral Extract from a Drupal Dump

What/why: Drupal dump extraction pipeline — the format, the whitelist, and why the pipeline is offline.

Purpose: turn a raw nanawall.com Drupal MySQL dump into a sanitized, neutral extract (a manifest plus per-table JSONL) that the bulk loader (#1097) can load into D1.

When to use: bootstrapping the full catalog from a fresh dump, or refreshing it — a newer dump re-run produces a new extract, which re-loads idempotently.

Prerequisites:

• A Drupal MySQL dump file, .sql or .sql.gz, obtained out of band. Never commit a dump to the repo — it contains PII until the extractor strips it.
• Node and the repo's dev dependencies installed (npm install).

Steps

1. Run the pipeline — one command reads the dump and writes the extract:

   npm run drupal:extract -- path/to/nanawall.sql.gz \
     --out extract \
     --drupal-version 10.3

   The dump is read directly as a stream: no MySQL server is started or connected to, and nothing goes over the network. A .gz dump is decompressed in-stream — no separate gunzip step.

2. Record the source provenance. Pass --drupal-version (it is not derivable from the dump). If the dump has no -- Dump completed on footer, also pass --dump-date <iso>. The run warns when either is missing — a provenance gap is never silent, because the #343 freshness view reports the dump's date, not the load time.

3. Read the run summary. It reports the source metadata, the whitelisted table and row counts, and how many non-whitelisted tables were skipped:

   extract complete in 12.4s → extract/
     source: nanawall.sql.gz (db nanawall, dumped 2026-07-01 03:14:15, Drupal 10.3)
     tables: 41 whitelisted, 58231 rows
     skipped: 96 non-whitelisted tables (names in manifest)

4. Confirm the sanitization. Open extract/manifest.json. skippedTables lists every excluded table by name — users, webform, commerce, session, cache, log, and config tables must all appear there, and none of them may appear under tables. The extractor enforces this via an explicit whitelist (never a blacklist), so an unknown future table is excluded by default; the guardrail spec fails the build if that ever regresses.

Output

extract/
├── manifest.json          versioned manifest + source provenance
└── tables/
    ├── node.jsonl
    ├── node__field_teaser.jsonl
    └── …                  one JSONL file per whitelisted table

The extract is the loader's input. Its format is versioned (formatVersion in the manifest) so the loader can refuse a shape it does not understand — see the extract format reference.

Loading the extract

The extract is loaded with the companion command (idempotent — re-running the same extract is a no-op):

npm run drupal:load -- extract            # local dev D1 (default)
npm run drupal:load -- extract --remote   # remote PROD D1 (nanaselect-db)

The load writes per-store counts, stamps every row with the dump's date as provenance, and records a bulk-load event the Drupal data screen reports as the catalog's last refresh. Since #1099 this pipeline is the ingestion channel — refreshing the catalog means re-running extract + load on a newer dump.

Local vs remote (--remote)

The flag changes only the write target — the source is the same neutral extract, and the planner/applier are identical, so all guarantees (idempotent, dump-date provenance, bulk-load event) hold on both targets:

	write target	execution path
(no flag)	local emulated dev D1	getPlatformProxy (local-only)
--remote	prod nanaselect-db	Cloudflare D1 REST API

A --remote load writes to production, so it is guarded (#1336):

1. It needs a Cloudflare API token with D1 edit access — set CLOUDFLARE_API_TOKEN (or CLOUDFLARE_D1_TOKEN). The account defaults to the one in wrangler.jsonc; override with --account-id or CLOUDFLARE_ACCOUNT_ID. The database id/name are read from wrangler.jsonc.

2. Before writing, it echoes the target (database name, id, account) and asks you to re-type the database name to confirm. A non-interactive run is refused unless you pass --yes (for CI):

   CLOUDFLARE_API_TOKEN=… npm run drupal:load -- extract --remote          # prompts
   CLOUDFLARE_API_TOKEN=… npm run drupal:load -- extract --remote --yes    # no prompt

Writes go through remote-sized atomic batches (D1's stricter per-request limits are respected), so a partial failure never leaves a half-written batch — and because the load is idempotent, a re-run cleanly resumes.

Safety

• The dump never enters a database. It is parsed as text; excluded tables are skipped without even parsing their values.
• The extract is safe to keep — it holds only whitelisted content tables. The dump itself is not; delete it once the extract is produced.