member-console/status/first-contact-ux-process.md

First-contact UX walk process (v1)

A repeatable method for evaluating the console the way a stranger meets it. Written 2026-08-22 for the M10 10g re-walk; versioned because the first run will teach us what to change. If a step earns its keep, keep it; if it produces noise, cut it in v2.

The problem this solves

Maintainers cannot judge first-contact UX. Whoever built a screen knows why it exists, what every label means, and what happens after every click, so nothing on it can confuse them. The two instruments this process uses to get around that are manufactured ignorance (evaluators who genuinely lack the project context, and are forbidden from acquiring it) and a written rubric (so every screen is scored against the same questions and findings are comparable instead of anecdotal).

Roles

Role	Who	Context they get	Context they must NOT get
Orchestrator	Full-context agent (maintainer's session)	Everything	—
Naive walker	Fresh agent, no project memory	README, a login URL, credentials, a persona mission	The codebase, internal docs, design docs, this process doc
Heuristic reviewer	Fresh agent	The rubric + one screen at a time	The codebase
Honesty auditor	Fresh agent	One screenshot + the matching model card from docs/models/	—
Outside model	Non-Claude model (e.g. via opencode)	Same as naive walker or reviewer, per lane	Same exclusions

The walker exclusion rule is the load-bearing part: confusion is the data, and a walker who peeks at the code stops producing it.

Phases

1. Build the yardstick

Research current practice (admin-console layout and prioritization, onboarding and empty states, progressive disclosure, form and microcopy design, heuristic-evaluation method itself) and distill it into a rubric: a short file of checkable questions applied per screen. The rubric must include:

• an information-architecture lens ("is this control on the right page at all?"), because misplaced controls masquerade as copy problems;
• an explicit anti-overwhelm score (choices visible per screen, what is above the fold, what is progressively disclosed), so "add more explanation everywhere" cannot become the default remedy;
• an honesty lens ("does the page imply anything the system does not actually do, or hide something it does?").

Output: the rubric file under status/.

2. Naive walkthroughs

Fresh agents walk the real running app in a browser, one persona mission each. Minimum personas: (a) an admin standing the console up for the first time, (b) a new member trying to accomplish the product's core promise, (c) an evaluator reading the README deciding whether to adopt. Each persona walks both an empty instance (true first boot; empty states are pure first-contact surface) and a populated one.

Walk protocol:

• The walker narrates the expectation before each click ("I expect this button to…"), so mismatches are caught, not just dead ends.
• Every question the walker asks aloud becomes a test: can the UI answer it within one click? If not, that is a finding.
• The walker keeps a friction log: where they hesitated, what they guessed wrong, what they wanted to know and could not learn.

Output: one friction log per walk, under status/.

3. Multi-model second opinions

Different models carry different priors; a screen that reads clearly to one family may confuse another. Outside models (via opencode or similar) run the text lane cold: read the README and setup docs as a stranger, critique page copy extracted as text, and, when the pipe supports images, score screenshots against the rubric. Claude agents keep the live-browser lane.

4. Honesty audit

For each major screen: one reviewer, one screenshot, the matching model card, one question — does this page tell the truth about what the system does? This phase exists because the model cards are the written ground truth for what each subsystem actually does, which makes the comparison mechanical instead of impressionistic.

5. Synthesize, triage, fix, re-walk

The orchestrator dedups and clusters all findings into a report with five buckets: copy fix, layout fix, missing explanation, structural IA question, works-as-designed. The maintainer triages the report; approved fixes become an OpenSpec change. The acceptance test is a literal re-walk: a fresh naive agent runs the same mission, and the previously logged friction points must not recur.

Method rules (added after the v1 Phase 1 research pass)

The 2026-08 research pass (four reports; distilled into ux-rubric-2026-08.md) hardened the method in six ways:

• Force the novice. AI walkers left to their defaults complete tasks better than real first-time humans and under-report learnability failures (CHI 2026, "Synthetic Cognitive Walkthrough"). The walker prompt must explicitly instruct confused-first-timer role-play; this is the single most load-bearing prompt line in the process.
• Walkers never see the rubric. The rubric is for heuristic reviewers and the orchestrator; a rubric-primed walker produces checklist answers, not genuine confusion.
• Evaluators are independent until synthesis. Pooling findings early anchors everyone on the first list and collapses the coverage benefit of multiple reviewers.
• Two passes per screen in heuristic review: free-form impressions first, then question-by-question; the second pass catches what the first glosses over.
• Severity discipline. Nielsen 0–4; record the observation first, assign severity in a separate pass; dedup by root cause, not reporter; triage must use all three buckets (fix-before-retest / track / won't-fix) — an empty won't-fix bucket means triage didn't happen.
• Stop condition is behavioral, not a headcount. Walk the planned persona × state matrix as the floor, then extend until two consecutive missions surface nothing new. The report must state the known blind spots of AI walkers (listed in the rubric) and the model versions used.

Standing guardrails

• Coverage is enumerated, not sampled. Pull the full route list from the router before the heuristic pass; the report states anything skipped.
• Serialize browser sessions. One debug browser instance means one walker at a time; walks are short, so this costs little.
• Reset state between walks. A walker inherits the previous walker's leftovers otherwise, and the "first boot" premise breaks.
• Findings cite the rubric question they fail, so triage argues about severity, not about whether something is a problem.

Run history

Run	Date	Scope	Outcome
v1, first run	2026-08 (M10 10g)	Whole console, operator + member	Evidence phases complete 2026-08-22: 8 missions, 48-screen dual-state sweep, 8 honesty audits, 3 heuristic reviews, 2 outside models, code verification. 19 deduplicated findings in ux-walks-2026-08/findings-2026-08.md; 6 environment artifacts caught by the verification pass (see that file's process record — v2 must flush Valkey + browser sessions on reset and pre-annotate seed infidelities). Triage pending.