nanaselect

Appearance

Media AI runs — description, alt-text & auto-classification (reference) ​

Issue #1243. Part of the media-curation workbench (#1238). Consumes #1113 (AI vision + embeddings) and the tag spine (#1112) — it does not build a parallel model.

Operator-triggered runs generate a description + accessibility alt-text and propose classification tags for media assets. Everything a run produces is a proposal: nothing takes effect without human acceptance, and every applied association is provenance-stamped.

What a run produces ​

OutputStoreGenerator
Proposed tags (system / project / attribute / factor / application)media_visual_tags (#1113)MediaAnalyzer (workersAiVisionAnalyzer #1188; stubAnalyzer)
Proposed description + alt-textmedia_visual_descriptions (#1243)MediaDescriber (workersAiVisionDescriber #1816; stubDescriber)
Similarity embeddingmedia_embeddings (#1113/#1189)Embedder

Tags are grounded in the controlled vocabulary (visualTagCandidatesFromOptions over mediaTagFacetOptions) — a run never invents free-form tags. Both the analyzer (#1188) and the describer (#1816) have real Workers-AI vision implementations selected by the #1190 egress gate (selectMediaAnalyzer / selectMediaDescriber, via the media-vision task profile #1418); with egress OFF the deterministic stubs run instead — same interfaces, no change to the lifecycle, review, or provenance below.

A describer may return null — "not enough signal to propose anything" — and the passes then persist nothing for that asset (#1606):

  • The stub describer proposes only from human-authored signal (title/caption): an asset whose only signal is its filename (or an existing alt-text) yields no proposal, never a filename echo dressed up with a confidence score.
  • The vision describer (#1816) proposes from actual image content (JSON-first reply, prose salvage, the LLaVA license-gate fallback captioner). An empty or unusable model reply proposes nothing, and #1780 filename-echo suppression applies to model output too — a reply that merely restates the filename is suppressed.
  • No silent zero (#1780/#1816): every zero-proposal path reports its reason through the run diagnostics (response + audit summary) — stub with no human-authored signal, unusable model text, unloadable image, or an idempotent skip (unchanged fingerprint / standing human decision).

Case-study grounding (#1782) ​

An asset carrying a project classification tag whose drupal_projects row links a case study (case_study_uuid) is analyzed with a bounded excerpt (≤1200 chars) of that case study's corpus content (summary/body/testimonial — PDF text included by the corpus ingest). The grounding:

  • joins the analysis fingerprint (analysisRef), so a re-run correctly re-analyzes when grounding appears or the case study changes;
  • feeds the stub analyzer's lexical signal, supporting proposals on descriptive axes (application, factor, attribute) — the project axis stays suppression-governed (#1781: the attached project is never re-proposed);
  • rides the vision prompt as a CONTEXT block that explicitly instructs the model to describe only what the image shows, informed by (not copying) the text.

Assets with no project tag or no linked case study behave exactly as before — identical fingerprints, identical proposals. Grounding ships inside the existing media-intelligence gates; no flag changes.

Run lifecycle ​

  • Tags + embeddings: runVisualAnalysisPass(db, analyzer, embedder, opts) (media-analysis-pass.ts).
  • Descriptions: runDescriptionPass(db, describer, { scanCap, coverageGapFirst }).
  • Shot-group-aware: runShotGroupDescriptionPass(db, describer) — describes the group's cover frame once and proposes that description to every member (the members are the same subject; #1240). A reviewer still accepts each.

Runs are:

  • Batched / throttled — scanCap (default 25000) bounds the assets scanned; the analysis pass also honours the #1191 model-budget governance.
  • Priority-ordered — the queue is gallery-order first (#1241), and coverageGapFirst sorts assets with no accepted description ahead of the rest, so a run fills coverage gaps first.
  • Resumable / idempotent — a proposal is keyed by the asset's analysis fingerprint (analysisRef); a re-run skips unchanged assets and never clobbers a human decision (an accepted/rejected row is left untouched). Re-run a partial run to resume.

Operational entry points: npm run media:describe [scanCap], npm run media:analyze.

Review loop ​

Project-wide bulk shot-group suggestion (#1374) ​

Scope the library to a project (the #1242 navigator) and Suggest shot groups runs the existing stem + near-dup clustering over ALL of the project's ungrouped images in one pass — bounded to the project's own asset ids, never a library scan. Candidates render with thumbnails, the backing signal (filename vs visually similar, plus visually confirmed when a filename cluster is corroborated by near-dup overlap — or filename only when embeddings haven't backed it), and near-dup tightness. Nothing is auto-created:

  • Confirm per candidate creates the group via the normal createShotGroup path (cover = first member; adjust cover/membership/variation on a member's Shot group tab, #1274).
  • Confirm all high-confidence creates the gate-passing batch in one action — visually-confirmed filename clusters and near-dup clusters at tightness ≥ 0.95.
  • A member consumed by an earlier confirm in the batch trips the already-grouped guard and the later overlapping candidate is skipped, never double-assigned.

On-demand runs (#1372) ​

Besides the whole-library pass, a curator can run analysis for one item from the per-asset editor's AI tab (/admin/media/[id]?tab=ai):

  • Run for this asset — re-analyzes the asset (force, so an unchanged fingerprint still re-runs) and proposes a fresh description, feeding the same review UI below the button.
  • Run for shot group (cover frame) — when the asset belongs to a shot group, the run-on-cover contract (#1373): the cover frame is analyzed once (one model call / one egress unit for the whole group — the token economics at 18.8k images), and the resulting tags + description are proposed to every member as proposed/ai with proposedForRef = group:<groupId>:<coverRef> — inspectably group-propagated. Per-member variation state (open/closed/day/night, #1274) is never touched; a member's accepted/rejected decisions are never re-proposed; stale group proposals from an earlier cover fingerprint are pruned. Escape hatch: a mis-grouped frame is detached on the Shot group tab (removeGroupMember) and re-run individually — it never silently inherits the group's classification.

Both scopes go through the SAME pipeline as the batch pass — the #1190 egress gate selects stub vs real for the analyzer and the describer (#1816), the #1191 monthly budget still applies (a budget-exhausted month refuses the run rather than bypassing governance; an egressed describe call is recorded to the egress audit trail and the spend ledger as pass media-describe), and every proposal is written proposed/ai with identical provenance.

The review surface is the per-asset edit page (/admin/media/[id], AI tab) and the media library (/admin/media, #1370 — card chips + bulk accept/reject; /admin/media-tags redirects there):

  • Description: accept — the description replaces the asset's caption and the alt-text replaces its alt_text (#1807; blank fields replace nothing, and the audit entry records the prior values so the overwrite is reviewable and manually revertible) — edit then accept (correct the text first; the edited text is what writes through), or reject.
  • Tags: accept (promotes to a live media_tags classification) or reject (remembered so a re-run won't re-propose it).

Nothing a run proposes is live until a human accepts it.

Provenance (#343) ​

Every proposal and decision is stamped:

  • source — ai (written by a run) → human (a rejection decision). An accepted tag proposal no longer flips in place: it is promoted to a media_tags classification row with source='ai' and the proposal is removed (#1805) — the accepting admin + date land in the audit trail.
  • status — proposed → rejected (remembered) or promoted away on accept (descriptions keep the in-place accepted state, #1243).
  • accepted_by + accepted_at — the reviewer identity and the date of the decision (NULL while proposed).
  • proposed_by_model — which model produced the proposal (e.g. stub-desc-v1), so a later model swap is auditable.

A rejected proposal is remembered by fingerprint, so the same run won't re-surface it — the human-correction loop is durable and feeds vocabulary learning.