member-console/docs/first-contact-rubric.md

title, audience, summary

title	audience	summary
First-Contact UX Rubric	developer	The scoring instrument the first-contact walk process applies: how to score, the A–G question bank, the walker protocol, the known blind spots of AI walkers, and the screen coverage checklist.

First-contact UX rubric (2026-08)

The scoring instrument for the first-contact UX re-walk, distilled from a four-report research pass (2026-08-22) covering admin-console layout, self-hosted first-run design, form/microcopy honesty, and evaluation methodology. Used by heuristic reviewers and by the orchestrator when mapping walker friction logs to findings. Naive walkers never see this file — a walker primed with the rubric stops producing genuine confusion.

Every finding must cite the question it fails (e.g. "fails C5, E7"). A finding that cites no question is an opinion; it goes to triage flagged as such.

How to score

• Two passes per screen: a free-form first-impressions pass, then a systematic question-by-question pass. The second pass catches what the first one glosses over.
• Reviewers work independently and never see each other's findings until synthesis (early pooling anchors everyone on the first reviewer's list).
• Record the raw observation first; assign severity in a separate pass.
• Severity on Nielsen's 0–4 scale: 0 not a problem, 1 cosmetic, 2 minor, 3 major (hard to recover from or recurring), 4 catastrophe (blocks the task). Judge it from frequency, impact, and persistence together, not gut feel.

Myth guards (do NOT file these as findings)

• More than 7 sidebar items is not a violation. The "7±2" menu rule misapplies working-memory research to recognition-based menus (NN/g repudiates it). Judge grouping quality instead.
• Content below the fold is not a violation. Users scroll when the layout invites it. Judge whether the top of the page establishes hierarchy and invites the scroll.
• A visible-but-disabled control is not a violation when it teaches a real prerequisite. Hiding is for irrelevant-to-role, not not-ready-yet (Nielsen's 2023–24 revision of the old hide-vs-disable rule).

A. Navigation and layout (every screen)

• A1. Are sidebar items grouped into labeled sections with scan-friendly ordering, rather than one undifferentiated list?
• A2. Are account/utility items (profile, sign-out) visually separated from task navigation?
• A3. Is navigation flat (one grouping level), never nested-doll?
• A4. Are nav labels named for the objects an operator manages, not for internal system or table names?
• A5. Does the screen have exactly one visually dominant primary action, with secondary actions clearly muted?
• A6. On detail pages, is the primary action docked in the same position across all detail-page types?
• A7. In dense tables, are row actions revealed on hover and focus (keyboard/touch parity), not persistent icon clutter — and not hover-only?
• A8. Do list pages show a scannable summary with drill-down to detail, rather than every column at once?
• A9. Are choices per screen reduced by sensible defaults rather than exposing every option every time?

B. Overview pages only

• B1. Does every tile pass the action test: can you name the operator action this number should trigger? (If not, it is a vanity metric.)
• B2. Is the page status-first (things needing attention) rather than raw-totals-first?
• B3. Roughly 3–7 tiles, most important first in reading order?
• B4. Does the overview link into sections rather than duplicating their data inline?
• B5. Do count tiles carry context (trend, delta, or a threshold) rather than a bare number?

C. First-run and empty states

• C1. Does the empty state say what this object type is and why the screen is empty, not just "no items"?
• C2. Does it offer exactly one primary next action?
• C3. No bare table shell with zero rows and no copy (looks broken, not empty).
• C4. No passive or blaming copy ("No results found").
• C5. Does the UI distinguish "nothing here yet — add one" from "you can't add one yet because X is missing"? Different states, different copy, different CTA.
• C6. Are empty states present even in low-traffic admin corners, not just the headline lists?
• C7. No forced linear wizard; orientation is a persistent, revisitable checklist or contextual hints. (Wizard acceptable only for irreducible global config.)
• C8. Is setup progress visible somewhere at a glance, rather than kept in the admin's memory?
• C9. Is a blocked control shown disabled with the reason at the point of the block, discoverable without hover (inline text or visible hint, since disabled elements often can't be hovered/focused)?
• C10. Does the blocking message name the specific missing prerequisite and deep-link to where to create it?
• C11. Is the dependency legible from both ends — the blocked screen says what's missing, and the prerequisite's empty state says what it unlocks?
• C12. On first login to an empty instance, is there a single unambiguous "first thing to do," not several blank sections competing?
• C13. Can a fresh admin get from first login to one concretely useful action with zero detours into external docs?
• C14. Is the setup dependency order stated up front somewhere in-product or in the README, rather than discovered by hitting blocked actions?

D. Forms and naming

• D1. Persistent visible label on every field (placeholder never the only label).
• D2. Placeholder text only as a format example.
• D3. Non-obvious fields carry visible help text near the field, not tooltip-only.
• D4. One marking convention for required/optional (mark the minority), applied identically across the whole app.
• D5. Do defaults pre-fill the choice most admins would make — and is a default on a consequential field visibly labeled as chosen, never silent?
• D6. No pre-selected default on a choice with irreversible or billing effects.
• D7. Does every error say what failed, why, and what to do next?
• D8. Errors appear only after the user acts (blur/submit), never while typing a first value.
• D9. Validation messages sit next to the field, not batched at the top.
• D10. One canonical name per concept, used identically on every screen where it appears (create form, list, detail, docs).
• D11. Is each domain term (pool, entitlement, ladder, claim…) defined in-place at its first appearance in a flow?
• D12. The plain-synonym test: could an everyday word replace this term without losing a real distinction? If yes, the jargon fails.

E. Consequence and honesty

• E1. Does the submit control or adjacent copy say whether the effect is immediate or queued/async?
• E2. Is irreversibility named explicitly in the confirmation ("cannot be undone"), never implied by a generic "Are you sure?"
• E3. Do high-risk confirmations require an affirmative step (retype, checkbox), not a default-focused OK?
• E4. Are billing/financial side effects stated before submit, not discovered after?
• E5. Do destructive dialogs label buttons with the actual verbs ("Delete domain" / "Keep domain"), never OK/Cancel?
• E6. Does async work say "queued/pending/processing" — never "saved" or "done" — until it actually settled?
• E7. Does every status vocabulary shown in UI have a visible failure state, or does "pending" silently cover both "working" and "broken"?
• E8. No fake-instant success for records the server must create (the ID doesn't exist until the server answers).
• E9. When the system can't verify or complete something, does the copy name the specific problem and the next step ("couldn't verify the DNS record; retrying in 5 minutes"), never a bare "something went wrong"?
• E10. Is success copy specific about what happened, not a "Success!" that could mean saved, queued, or started?

F. Information architecture (screen and flow level)

• F1. The misplaced-control test: is this control on the right page at all? Would a newcomer look for it here first?
• F2. Do the operator and member surfaces have genuinely separate IAs sized to their task counts, not one tree with items permission-hidden?
• F3. Where settings exist at different scopes (instance-wide, per-organization, per-provider), is scope an explicit grouping axis?
• F4. Flow-level consistency: does a concept keep the same name, position, and behavior across every step of a multi-screen flow?

G. Flow-level walkthrough questions (per mission step)

The cognitive-walkthrough four, asked at every step of a mission:

• G1. Would a first-timer form the right goal at this point?
• G2. Would they notice the correct control is available?
• G3. Would they connect that control to their goal from its label and placement alone?
• G4. After acting, does feedback clearly show progress toward the goal?

Any "no" is a logged finding at that exact step.

Walker protocol (naive walks)

1. Force the novice. The walker's prompt must instruct it to role-play a confused first-time user. Published result (CHI 2026 synthetic cognitive walkthrough study): LLM walkers left to their defaults complete tasks better than real novices via more optimal paths and under-report learnability failures — the exact signal we need.
2. Expectation before action. Before every click: "I expect this to…" — then act, then record match/mismatch. This operationalizes G3.
3. Neutral self-prompts only. "What am I thinking? What do I want?" — never "did I notice X?", which telegraphs the answer.
4. Concurrent narration plus a short retrospective debrief at mission end: what was hardest, what would you tell the next new user, what do you still not understand.
5. Every unanswered question is data. If the walker wants to know something and the UI can't answer it within one click, log it.
6. Friction log entries: step, expectation, what happened, confusion (if any), question asked (if any). No rubric references — the orchestrator maps those at synthesis.

Stop condition and triage rules

• Mission floor: the planned persona × state matrix. Stop extending only after two consecutive missions surface nothing new (the classic "5 users find 85%" rule is contested; the stop signal is behavioral).
• Dedup by root cause, not by reporter: the same issue found by three evaluators is one finding with elevated frequency.
• Triage buckets: fix-before-retest (sev 3–4), track-don't-block (sev 2), won't-fix-accepted (sev 0–1). A triage with an empty won't-fix bucket hasn't triaged.
• The acceptance re-walk targets the original sev 3–4 findings: each must be demonstrably gone in the re-walked mission, not just "code changed nearby."

Known blind spots of AI walkers (state these in the final report)

1. Default over-competence (countered by protocol rule 1, but never fully).
2. No affect: agents don't get frustrated or give up, so abandonment signals — the strongest human severity markers — never appear.
3. Motor/visual blindness: agents read the DOM, not pixels; tiny targets, low contrast, and scroll confusion may not register.
4. Literal-text bias: over-flags wording a human would resolve from layout at a glance; under-flags visual-hierarchy problems.
5. Manufactured homogeneity: repeated runs of one model are one persona repeated, not population variance — cross-mission agreement is weaker evidence than it looks. (Partly countered by the multi-model lane.)
6. Model dependence: findings are tied to specific model versions; the report names them.

Coverage checklist

Every screen below gets: a heuristic pass (sections A–F), an honesty-audit pass where a model card covers it, and walk coverage via at least one mission. Fill columns during the run; anything skipped is stated in the report. Empty-state column: walk on a fresh instance; n/a where the screen cannot exist empty (e.g. a person detail page).

Screen	Route	Heuristic	Honesty audit (card)	Walked	Empty state
Member dashboard	/		identity-organization-workspace
Member products	/products		product-catalog
Member billing	/billing		payments-billing
Member domains	/domains		domains-registry
Public domain ask	/domains/ask		domains-registry
Checkout flow	POST /billing/checkout → Stripe		payments-billing		n/a
Login flow	Keycloak redirect		—
Operator overview	/operator		— (all cards)
Operator products	/operator/products (+ detail)		product-catalog
Operator plan ladders	/operator/plan-ladders (+ detail, validation)		plan-ladders-transitions
Operator plan topology	/operator/plan-topology		plan-ladders-transitions
Operator org types	/operator/org-types		identity-organization-workspace
Operator organizations	/operator/organizations (+ detail)		identity-organization-workspace
Operator person detail	/operator/persons/{id}		identity-organization-workspace		n/a
Operator grants	/operator/grants		entitlements
Operator entitlement sets	/operator/entitlement-sets (+ detail)		entitlements
Operator billing: accounts	/operator/billing/accounts		payments-billing
Operator billing: subscriptions	/operator/billing/subscriptions		payments-billing
Operator billing: invoices	/operator/billing/invoices		payments-billing
Operator billing: payments	/operator/billing/payments		payments-billing
Operator domains	/operator/domains		domains-registry
Operator integrations	/operator/integrations (+ settings)		provider-integration
Error pages	404 / 500		—		n/a

Sources

NN/g (ten heuristics, cognitive walkthroughs, severity ratings, empty states, error messages, hide-vs-disable revision, magic-number-7, flat-vs-deep); MeasuringU (heuristic-eval effectiveness, ~30–40% overlap with behavioral testing); Faulkner and Spool/Schroeder (five-user critique); Ericsson & Simon (think-aloud validity); Baymard/SubUX/Balsamiq (forms, button hierarchy); Pencil & Paper (enterprise tables/UX); Stripe Dashboard teardowns; Linear settings redesign; Discourse admin quick-start; Home Assistant onboarding; GitLab admin empty-states epic; Smashing Magazine (disabled buttons, confirmation dialogs 2024); Atlassian design content guide; UXAgent (arXiv 2504.09407); Synthetic Cognitive Walkthrough (arXiv 2512.03568, CHI 2026); Lost in Simulation (arXiv 2601.17087).