NanaSelect operations runbook

Purpose: the one page for operating NanaSelect — starting and stopping the dev environment, deploying, finding logs, managing the database, and recovering from the failures we actually hit. Deeper procedures live in docs/how-to/; this page tells you which lever exists and where it is.

When to use: day-to-day operation, deploys, and any "the site is doing something weird" moment.

What's running where

Piece	What it is
nanaselect Worker	The app (SvelteKit → Cloudflare Worker), account paul@iwpi.com. Serves https://nanaselect.iwpi.com (custom domain, behind Cloudflare Access — see admin-access / poc-access). The *.workers.dev and preview hosts are deliberately disabled (Access can't gate them).
content-refresh Worker	Cron worker (workers/content-refresh/) — scheduled content refresh.
session-purge Worker	Cron worker (workers/session-purge/) — session retention purge (#347).
D1 nanaselect-db	The product database (binding DB).
R2 nanaselect-corpus / nanaselect-media	Corpus documents and media objects.
Vectorize + Workers AI	Retrieval/embedding for the intelligence features — all behind OFF-by-default flags (intelligence rollout).
Docs site	https://docs.nanaselect.iwpi.com — see building-the-docs-site.

Required Worker config (CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUD, DRUPAL_BASE_URL, session TTLs, …) is committed in wrangler.jsonc vars. Secrets are not in the repo — inspect with wrangler secret list, set with wrangler secret put <NAME>.

Dev environment

dev-up "$(pwd)" --verbose     # migrations → seed → server → health (reads dev.json)
dev-up "$(pwd)" --check       # health probe only
npm run dev -- --port 6650    # bare dev server, if you don't want dev-up

• Local URL: http://localhost:6650 · health endpoint: GET /health.
• Admin sign-in is automatic locally (dev bypass as admin@test.local); other roles via .dev.vars — see admin-access.
• Stop: kill the vite dev process (dev-up runs it in the foreground session; pkill -f "vite dev" if orphaned).
• Seeding is npm-script-driven, not dev-up's per-file runner: npm run seed:local (build → seed/seed.sql → catalog fragments → media objects). Both seed paths apply the fragments through one runner (scripts/seed-fragments.mjs); npm run seed:fragments:local re-applies just those. If a reset leaves the enum catalog empty, the fragments did not run — that was bug #1885.

Recovery — dev server unhealthy: dev-up "$(pwd)" --verbose (it restarts and re-checks). After db:reset:local, always restart the dev server — it keeps a handle on the deleted D1 file, so every request 500s until it is restarted (/health still reports healthy, so trust the app, not the probe). Local D1 wedged or FK-broken: npm run db:reset:local (drops .wrangler/state/v3/d1, re-applies migrations, re-seeds). Both are destructive only to local state.

Deploying

npm run deploy            # config gate → migration gate → build → wrangler deploy (the app Worker)
npm run db:apply:remote   # remote D1 migrations — run BEFORE a deploy that needs them
npm run deploy:refresh    # content-refresh cron worker
npm run deploy:purge      # session-purge cron worker

npm run deploy runs two blocking gates before it builds — fix what they name; don't bypass either:

1. Config gate (scripts/check-deploy-gate.ts, #1058) — the admin fail-closed invariant in wrangler.jsonc (Access team domain + AUD present, no ADMIN_DEV_BYPASS). Network-free.
2. Remote-migration gate (scripts/check-remote-migrations.ts, #1685) — refuses to deploy when remote D1 is behind the append-only migration set, so code can't ship ahead of the schema it reads (the #1682 prod-500 class). If it blocks, it names the pending migrations; the remedy is to run npm run db:apply:remote first, then re-deploy. The gate never auto-applies migrations — schema promotion stays a deliberate, reviewable step. It fails closed: an unreadable wrangler d1 migrations list result blocks rather than assuming the remote is current.

Prerequisite: wrangler login (or CLOUDFLARE_API_TOKEN) for the paul@iwpi.com account (the migration gate also needs it — it queries remote D1).

Activating the gated intelligence features (Vectorize/R2 prerequisites, smoke checks, flag flips, rollback) is its own sequence: deploy-and-activate-intelligence. Each flag has a one-flag rollback — that doc is also the recovery path if an intelligence swap misbehaves in prod (flip it off, redeploy).

Logs & observability

wrangler tail nanaselect                                        # live prod logs
wrangler tail --config workers/session-purge/wrangler.jsonc     # cron workers
wrangler tail --config workers/content-refresh/wrangler.jsonc

Dashboard: Cloudflare → Workers & Pages → nanaselect (metrics, errors, cron run history). In-app: /admin/audit (who changed what) and /admin/analytics.

Database operations

npm run db:generate       # new migration from schema change (drizzle-kit)
npm run db:apply:local    # apply locally
npm run db:apply:remote   # apply to prod D1
npm run db:studio         # browse the local DB
npm run seed:remote       # seed/refresh remote catalog (FK-ordered fragments)

Migrations are append-only in drizzle/; never edit an applied one — add a new one.

Table rebuilds can destroy prod data that local apply preserves (#1888).PRAGMA foreign_keys=OFF — the pragma drizzle emits around every table rebuild — does nothing on remote D1: D1 runs each migration inside a transaction and SQLite silently ignores the pragma there, so DROP TABLE fires ON DELETE cascade on every FK child as though the parent rows were deleted. Miniflare's local D1 does not enforce this, so a clean local apply is not evidence a rebuild is non-destructive (migration 0093 deleted all 69 attribute_enum_value_aliases rows on prod while local kept every one; repaired by 0094). Two guards enforce this:

• npm run db:generate (and npm run db:check-safety standalone) refuses a generated migration that drops a table other tables REFERENCE, unless the same migration backfills each cascade child or carries an explicit -- cascade-ack: <table> — <why> comment.
• npm run db:apply:remote (scripts/apply-remote-migrations.ts) re-runs that check on the pending set, snapshots row counts for every FK child of a rebuilt table before applying, and re-counts after — a cascade loss fails the apply loudly instead of surfacing in a later audit. On a reported loss, restore via D1 Time Travel or re-seed the named table immediately. Remote data procedures (Drupal → NanaSelect) are documented in managing-drupal-data (drupal:extract / drupal:load) and refresh-the-nanasage-corpus-from-a-drupal-dump (corpus:ingest / corpus:embed).

Backup & restore (#1960)

npm run backup:remote     # timestamped full remote-D1 SQL export + fixture refresh

One command, two artifacts:

• SQL export — backups/nanaselect-db-<UTC-stamp>.sql (git-ignored), a full wrangler d1 export of the remote database.
• Selection-model snapshot — refreshes src/lib/server/__fixtures__/recommendation-parity.json via the existing fixture:recommendation path, so the model state is diffable in git.

The command is admin-tier: it fails loud and refuses to run if the wrangler auth check or the target-database check fails — fix auth (npx wrangler login) and retry; never work around it.

Restore path 1 — D1 Time Travel (point-in-time, preferred when inside the window). Retention is 30 days on Workers Paid (this account's plan; 7 days on Free). Find a restore point near the incident, then restore:

npx wrangler d1 time-travel info nanaselect-db --timestamp "<ISO-8601>"
npx wrangler d1 time-travel restore nanaselect-db --timestamp "<ISO-8601>"

Restoring rewinds the whole database to that moment — anything written after the timestamp is lost. Time Travel also records a bookmark before the restore, so a restore is itself reversible.

Restore path 2 — import the SQL export (when the incident is older than the retention window, or you want exactly a known backup). The export contains CREATE TABLE + data for the full database; import into a fresh or emptied database to avoid table-exists conflicts:

npx wrangler d1 execute nanaselect-db --remote --file backups/nanaselect-db-<stamp>.sql

After either restore, run the standard verification (below) plus npm run deploy:migrations-check — a restore to an older point can leave remote behind on migrations.

Common failures

Symptom	First move
Local dev 500s / dead port	dev-up "$(pwd)" --verbose — restarts + health-checks.
Local DB errors after pulling migrations	npm run db:apply:local; if still broken, npm run db:reset:local.
Deploy blocked	Read the gate output — the config gate (npm run deploy:gate) or the migration gate (npm run deploy:migrations-check) names the failing condition. "Remote D1 is N behind" → npm run db:apply:remote, then re-deploy.
Prod /admin loops to login or 403s a staff member	Two-part grant: Access policy and /admin/users roster must both include the email — admin-access.
Prod error spike	wrangler tail nanaselect while reproducing; check the dashboard error graph for the deploy boundary; if it correlates with an intelligence flag flip, flip it back (rollback).
Cron worker didn't run	Dashboard → the worker → Triggers → cron history; re-test locally with npm run purge:dev / npm run refresh:dev (--test-scheduled).
OTP mail not arriving	Zero Trust dashboard → Access → the login attempt log; confirm the email is in the Access policy.

Verification after any operation

curl -fsS https://nanaselect.iwpi.com/health   # prod (needs an Access session/service token if gated)
curl -fsS http://localhost:6650/health          # local
npm test && npm run check                       # the suite the deploy gate leans on