Pre-handover admin usability walkthrough — protocol

Issue: #1526 · Part of #1519 (top-tier admin experience). This session is the handover gate: its go/no-go note decides whether /admin is ready for non-developer NanaWall admins (~2026-07-23).

Objective

Watch at least one real future admin (not Paul) complete the core admin tasks unaided, and answer: can a first-time NanaWall admin operate the knowledge base without a developer explaining the vocabulary? (#1519 success metric). Every struggle is captured, triaged, and either fixed before handover or filed.

Participants

• Number: 1 minimum (2–3 preferred). Must be a genuine future admin — someone who will actually manage systems/attributes/media/content after handover. Paul is the moderator, never the participant.
• Tech level: whatever they really are — do not pre-train them beyond the setup below.

Setup (before the session)

1. Environment: the gated review host https://nanaselect.iwpi.com (real deployed Worker behind Cloudflare Access — not the dev bypass, so the session exercises the true login).
2. Access: add the participant's email to the Access policy in the Zero Trust dashboard, and roster them at /admin/users (role admin for the session; see admin-access.md).
   • Optional two-birds step: have them visit /admin after the Access policy admits them but before rostering — the expected 403 page live-verifies the un-rostered negative path (todo #1557). Then roster them and continue.
3. Recording/notes: screen share + the per-task notes template below. One note-taker (or record and transcribe) — the moderator should not be heads-down writing.
4. Confirm the OTP login works for the participant's email before the session (a bounced OTP mail burns the slot).

Moderator script (think-aloud)

"As you work through these tasks, please think out loud — what you're looking at, what you're trying to do, what you expect to happen. There are no wrong answers; we're testing the design, not you. If something is confusing, that's exactly the feedback we need. I can't help during a task, but you can give up on one at any time — giving up is a valid result."

During tasks prompt only with: "What are you thinking?", "What do you expect to happen?", "What are you looking for?" Never lead ("you probably want that button"), explain ("that's actually how X works"), or rescue early.

Tasks (7 core)

Each task is read to the participant verbatim. Unaided means no moderator hints; consulting the in-app help/docs links counts as success, not a struggle.

#	Task (read aloud)	Success criteria
1	"A buyer clicking the ? next to the Geography question should see better explanatory text. Update that question's education content, and move the question one step earlier in the flow."	Edits the factor's education text and its placement/order on /admin/factors, saves, and states where the change will show up.
2	"Add this photo to the SL70 system and make sure it will actually show on the public site."	Attaches the image via the media widget, notices/clears the rights posture, and verifies it renders publicly (e.g. opens the public page or reads the rights indicator correctly).
3	"When a buyer answers Coastal for Geography, that should map to the Windloads attribute with a value the recommendation can use. Add that mapping."	Creates an option → attribute mapping with the expected value on the factor's Options tab, and can say what it does.
4	"The AI has suggested tags for corpus content. Review one suggestion — accept it if it's right, reject it if it's wrong."	Finds the review queue (/admin/content-tags or the dashboard action item), inspects a suggestion, and accepts/rejects with a reason they can articulate.
5	"On the guided-flow results, one system shows a photo you don't recognize. Find out why the app chose that image."	Traces the image to its source (illustration assignment / attachment / canonical flag) using admin surfaces — not guessing.
6	"It's Monday morning. Show me how you'd find out what needs your attention today, and handle one item."	Uses the dashboard work-queue ("Things to action"), picks an item, and lands on the right screen to resolve it.
7	"Your colleague Dana will help with content editing, but must not be able to change recommendation rules. Set up their access."	Adds the email at /admin/users with role data_editor and states (or discovers via help text) that the Access policy must also admit the email.

Per task record: outcome (success / success-with-struggle / gave-up), time, and at least one verbatim quote.

Session notes template (one per task)

### Task N — <name>
- Outcome: success | struggle | gave up
- Time: <mm:ss>
- What happened: <observed path, wrong turns, where they looked first>
- Quote: "<verbatim>"
- Severity if struggle: critical | high | medium | low

Severity scale — critical: prevents task completion; high: significant delay/frustration; medium: minor confusion; low: cosmetic.

Post-test questions

1. "Which task felt hardest, and why?"
2. "Is there a word or label in the admin that doesn't mean anything to you?"
3. "If a change you make is wrong, how confident are you that you could undo or fix it?"
4. "Anything you expected to be able to do that you couldn't find?"

Triage & fix cycle (after the session)

1. Compile findings into the severity table (finding / severity / frequency / recommendation).
2. Critical + high = handover blockers: fix before handover (file as bug or req with approved + p2-high, reference #1526).
3. Medium/low: file as issues (label by area), no gate.
4. Post the go/no-go note on #1519:

## Walkthrough go/no-go (#1526)
- Participant(s): <who, role>
- Tasks: <N>/7 completed unaided; <N> with struggle; <N> gave up
- Blockers found: <list or "none"> — fixed in <commits/issues>
- Verdict: GO / NO-GO for handover, because <one sentence>

Go bar: every task completed (unaided or with self-recovered struggle), and zero unfixed critical/high findings.