Props and Pilots A to Z

Randomly assigning live users to two or more versions of a product and comparing a predefined outcome to see which performs better. A multivariate test varies several elements at once.

Also called split test · multivariate test (variant)

Drafting assessment questions and tasks to match an assessment blueprint, so that each item measures its intended outcome at its intended level.

Also called item development

Testing key tasks using assistive technologies, such as screen readers, magnifiers, switch access and voice control, ideally with people who use them every day.

Also called screen reader testing

Releasing a near-final product or feature to a selected group of users before general release, to find problems and gather feedback in real use.

Also called early access programme · pilot release

A map of the decision points, choices, consequences and endings in an interactive scenario, planned before it is written or built.

Also called decision tree map

Linked screens with clickable hotspots that simulate how a product will work, made in a prototyping tool without code.

Also called click dummy · interactive prototype · high-fidelity prototype

Interviews in which respondents answer draft survey or test questions while explaining how they understand each one, to find wording problems before use.

Also called cognitive pretesting

An expert review in which evaluators step through a task as a new user would, asking at each step whether the user would know what to do and see that it worked (Wharton et al., 1994).

Delivering a service manually, in person, to a few users, to learn what they value before automating it. Unlike Wizard of Oz, users know a person is doing it.

Also called concierge MVP

A prototype built with real or realistic draft content instead of placeholder text, so that structure is designed around what the content needs.

Also called content prototype · proto-content

A sketch or prototype of a data dashboard, ideally using real or realistic data, to test whether it answers its users' questions.

Also called data prototype

A research approach in which researchers and practitioners design an intervention, test it in real classrooms over several cycles and refine both the design and the theory behind it (Brown, 1992; Collins, 1992).

Also called DBR · design experiments

A small-scale model of a service, built on a tabletop with figures and props, through which a team walks the service step by step (Stickdorn et al., 2018).

Also called tabletop walkthrough · service model

A review of content by subject experts for accuracy, currency and appropriateness, separate from usability testing.

Also called subject-matter expert (SME) review

A map on which a team plots the prototypes it has built along two chosen dimensions, to see which areas have been explored and which have not.

Offering a button or link to a feature that does not yet exist, to measure how many people try to use it. Those who click see an honest message that the feature is being considered.

Also called painted door test

A four-part grid for recording feedback during or after a test: likes, criticisms, questions and ideas.

Also called feedback grid

Showing users a screen and a task, and recording where they click first. A correct first click strongly predicts task success.

Showing a screen for five seconds, then asking what users remember and what they think it is for, to check the first impression and main message.

Quick, informal tests of five to ten minutes with whoever is available, in a public place or office, to catch obvious problems cheaply.

Also called hallway testing · corridor testing

Experts review an interface against a set of usability principles, most often Nielsen's ten heuristics, and rate the severity of each problem.

Also called expert review · usability inspection

A three-prompt format for giving feedback: what I like, what I wish were different, and what I wonder about.

Also called rose · thorn · bud (a similar format)

Analysing how each question performed: its difficulty (the proportion correct), its discrimination (whether stronger students got it right more often) and how each distractor was chosen.

Also called item statistics

A draft version of learning materials, such as worksheets, slides or an online module, made quickly to try with learners before full production.

Also called draft module · pilot materials

Teaching a short segment of a lesson to peers, or to a small group of students, to test it and get feedback before teaching the full lesson. Microteaching was developed at Stanford in the 1960s.

Also called microteaching · rehearsal lesson

The smallest version of a product that lets a team learn whether its core idea works with real users (Ries, 2011). An MVE extends the idea to the smallest set of partners needed to test an ecosystem. For the term itself, see Stacks and Sprints B9.

Also called MVP · minimum viable ecosystem (MVE)

A facilitator watches a user attempt realistic tasks with a product or prototype, asks them to think aloud, and records where they succeed and struggle.

Also called usability testing · user testing · structured usability testing

A process in which markers mark the same samples, compare and discuss their judgements, and agree a shared standard before and during marking.

Also called rubric calibration · standardisation · marker calibration

A game made with cards, boards, dice and tokens to test rules and play before building a digital version.

Also called tabletop game prototype

Hand-drawn or printed screens and interface pieces that a person "operates" by swapping paper in response to a user's taps and choices.

Also called paper prototyping

A small-scale trial of a programme, product or policy in real settings, to learn whether it works and how to implement it before wider rollout.

Also called trial · field trial · pilot lesson

Testing a policy idea by making the artefacts people would actually meet, such as a letter, a form, a guideline or a service step, and trying them with users before the policy is finalised.

Also called prototyping policy

Measuring learners' knowledge or skill before and after an intervention, ideally with a comparison group, to estimate learning gain.

Also called learning gain measurement

Showing users design options and asking which they prefer and why, or asking them to pick words from a list, such as Microsoft's 118 product reaction cards, to describe a design (Benedek & Miner, 2002).

Also called product reaction cards · desirability study

A planned succession of prototypes from Stanford's ME310 course, each answering a different question, from whether a key experience matters to whether the whole solution works.

Also called ME310 prototype types: critical experience · critical function · dark horse · funky · functional · X-is-finished · final

Checking that users understand text. A cloze test removes every fifth to seventh word and asks readers to fill the gaps. Scores of about 60% or more suggest the text is readable for that audience.

Also called cloze test · comprehension testing

A form of usability testing in which the team fixes clear problems between sessions, so later participants test an improved version (Medlock et al., 2002).

Also called rapid iterative testing and evaluation

Writing the criteria for judging a performance and the descriptions of each level of quality.

Also called marking scheme design · scoring guide

Acting out a service at full scale, with staff and props, in the real or a simulated setting, to test and refine the interactions.

Also called service prototyping · service staging · investigative rehearsal

A hierarchical diagram of the pages or sections of a product, showing how they are grouped and linked.

Also called navigation design · IA diagram

An interview in which a proposed solution, often a prototype or demo, is shown to users to check whether it solves their problem and whether they would use it (Maurya, 2012).

Also called problem and solution interview

Trying out a room layout at full scale with tape on the floor, cardboard or movable furniture, before committing to a design.

Also called full-scale mock-up · tape-out · pop-up classroom

A one-page plan for a single test, stating the assumption, what will be done, what will be measured and what result would count as success. Strategyzer's test card and the Toolbox's testing sheet are versions of this.

Also called testing sheet · prototype to test · test plan

Asking participants to say what they are thinking as they work, so that researchers can hear expectations, confusion and reasoning.

Also called concurrent think-aloud

Participants find items in a text-only version of a site's hierarchy, which tests the structure and labels without the influence of visual design.

Also called reverse card sort

Participants complete tasks on their own, usually through an online tool that records their screen, clicks and sometimes voice.

Also called remote usability test

A diagram of the steps and decisions a user goes through to complete a task, from entry point to goal. A wireflow uses small wireframes as the steps.

Also called task flow · wireflow · screen flow

A short film showing a concept in use, made with phone footage, sketches or mock-ups, to communicate an idea and gather reactions before building it.

Also called vision video · concept video

Drawings of a screen's layout, content and functions without visual styling, used to agree structure and behaviour.

Also called wireframes · low-fidelity mock-ups

A prototype in which a hidden person produces the system's responses, so that users experience a working product before the technology exists.

Also called Wizard of Oz testing