Quality Eval Harness

How NanaSelect proves that activating a real AI model is genuinely better than the deterministic baseline — not just more expensive (issue #1192, part of capability #1185). Each of the three intelligence swaps is gated by a measured metric against a documented go/no-go bar, so a swap goes live only on a real lift and a regression is caught. It is the go-live evidence for RAG Intelligence Bindings, priced under Model Budget Governance, and its bars feed the feature-flag rollout (#1193).

The one-line version: for each swap, the harness measures the real-model metric vs the deterministic-stub baseline over a versioned eval set, and a pure gate promotes it only when it clears an absolute floor AND beats the baseline by a documented margin — otherwise the deterministic path stays live.

The three swaps + their metrics

Swap	Baseline (deterministic)	Candidate (real)	Metric
Corpus retrieval (#1187)	lexical scoring	hybrid lexical + Vectorize (RRF)	recall@k
Visual tags (#1188)	stubAnalyzer (lexical)	a real vision MediaAnalyzer	precision/recall/F1
Media similarity (#1189)	brute-force D1 cosine	Vectorize ANN	precision@k

Metrics are pure, set-based functions in src/lib/server/eval/metrics.ts (recallAtK, precisionRecallF1, precisionAtK), unit-tested offline.

The eval sets (AC-01)

Versioned JSON under data/eval/, each carrying its cases and its go/no-go bar:

File	Swap	Cases	Ground truth
corpus-retrieval-eval.json	corpus-retrieval	6	semantic-paraphrase queries → seeded corpus item ids
media-tag-eval.json	visual-tags	4	curated media_tags on the seeded Cero II assets
media-similarity-eval.json	media-similarity	3	same-system neighbours in the seeded Cero II cluster

A set is validated on load by parseEvalSet (src/lib/server/eval/run.ts) — a malformed fixture fails loudly rather than skewing a metric. Eval sets are a reviewable, versioned quality contract; extend them as the corpus + media library grow.

The go/no-go gate + bars (AC-04)

evaluateSwap(baseline, candidate, bar) (src/lib/server/eval/gate.ts) is the pure promotion decision. A swap is GO only when BOTH hold:

• floor — candidate ≥ bar.minCandidate (an absolute usable-quality bar), and
• lift — candidate − baseline ≥ bar.minLift (a real improvement over the baseline).

Either miss ⇒ NO-GO, and the deterministic path stays live (this is exactly what keeps DEFAULT_HYBRID/DEFAULT_MEDIA_SIM OFF until a swap earns its flip — the reversibility contract of #1193).

Drafted bars (decision #1195). These are conservative first cuts from the deterministic baselines. Paul reviews the numbers and holds the prod-flip gate — the harness reports the verdict, a human authorizes the flip.

Swap	Metric	minCandidate	minLift	Rationale
corpus-retrieval	recall@8	0.80	0.05	lexical already recalls strongly — no regression, recover ≥1 missed case
visual-tags	f1	0.50	0.10	a real model must clear usable precision/recall + clearly beat the lexical stub
media-similarity	precision@12	0.50	0.10	"assets like this" must be right more often than the brute-force stub-vector baseline

They live in DEFAULT_EVAL_BARS and in each eval set's bar field; tune them in the fixture/constant (a quality contract), not via env (env is for runtime ops).

Running the harness (AC-03)

Offline / CI — the whole scoring path (fixture → metric → aggregate → gate) runs with zero bindings:

npm run test:unit -- src/lib/server/eval

src/lib/server/eval/harness.spec.ts validates all three fixtures and runs the visual-tag baseline end-to-end through the deterministic stub — no D1, no Workers-AI, no Vectorize. This is the CI regression guard.

Live operator harnesses — measure the real candidate against the baseline (these need live infra, like the embedding backfills):

npm run corpus:eval   # recall lift, lexical vs hybrid (needs the corpus index)
npm run tag:eval      # visual-tag precision/recall/F1 (stub baseline)
npm run sim:eval      # media-similarity precision@k, brute-force vs Vectorize

Prerequisite (media swaps): the media harnesses (and media:analyze/media:embed) require the nanaselect-media-sim Vectorize index to exist, or getPlatformProxy fails at init:

wrangler vectorize create nanaselect-media-sim --dimensions=768 --metric=cosine
npm run media:embed    # populate embeddings before sim:eval yields signal

(Tracked as ops TODO #1218.) Each live harness prints the per-case metrics and the final GO / NO-GO verdict.

How it gates rollout (#1193)

The eval bar is the thing that flips a swap's default-OFF flag. The rollout (#1193) reads a swap as promotable only once its harness reports GO and Paul authorizes it; below the bar, the feature flag stays OFF and the deterministic path serves — no silent regression reaches a buyer.