Gerald Ajam

Designing questions that reveal thinking

Most questions are built to be marked. They separate right from wrong, and they do it well. A question built to show thinking has another job. Each possible answer tells the teacher something, and what happens next is decided before anyone answers.

Status Position

Published November 2026

All notes

In short

A question shows how a learner is thinking only when it is designed to, with wrong answers built from real misconceptions and a request for the reasoning behind the answer.

The shift it turns on

The usual reading

A question that checks

Right or wrong, then a score.

The reading it moves to

A question that diagnoses

Every answer points to a way of thinking, and to what happens next.

01The situation

Right or wrong, and little else

Learning platforms ask a great many questions. Most are marked automatically, and most return one bit of information: right or wrong. A dashboard turns those bits into a percentage, and the percentage goes into a report.

A percentage correct hides most of what a teacher needs. It does not say who was wrong or which wrong answer they chose. A class at seventy per cent might share one misconception or hold six different ones, and those call for very different lessons. Right or wrong is just as thin for the learner. Van der Kleij and colleagues (2015) pooled studies of feedback in computer-based learning and found that being told only whether an answer was correct did almost nothing. An explanation did far more.

And where a polished answer can be produced without the thinking behind it, a question that checks only the answer checks very little. Dawson and colleagues (2024) argue that validity matters more than cheating: what counts is whether the evidence shows what this learner can do.

02The tension

What would each answer tell you?

What if a question were judged by what its wrong answers tell you? Black and Wiliam (1998) called assessment formative when its evidence is used to adjust the teaching. That sets a demanding standard for a single question. Before it is asked, the person asking should know what each likely answer would make them do.

Most questions fail that test quietly. “What did you learn today?” accepts any answer, so no answer changes anything. A multiple-choice item with wrong options there to fill the slots shows only that someone was wrong. The thinking is in the error, and the question was never built to catch it.

Before a question is asked, the person asking should know what each likely answer would make them do.

03The move

Build the question from the mistakes

  1. Start from the misconceptions

    Write down two or three ways learners typically get this idea wrong, from research on the topic or last year’s marking. Then write one wrong option for each, exactly what a learner holding that idea would choose. Sadler (1998) made the case for science tests whose wrong answers come from documented student ideas, so that a wrong answer says which idea a learner holds. A learner who expands 3(x + 4) as 3x + 4 and one who writes 3x + 7 have made different errors, and each needs a different response.

  2. Decide before you look

    Write the decision rule beside the question. “If more than a third choose B, I go back to the area model. If most are right and a few choose D, I take that group aside.” A rule written after the results are in bends to fit them. Wiliam calls this decision-driven data collection: the decision comes first, and the data is gathered to make it.

  3. Ask how sure they are

    Pair the answer with a confidence rating, and keep it ungraded. The pair separates four cases a score merges. Sure and right can move on. Unsure and right needs practice. Unsure and wrong needs teaching again. Sure and wrong is a misconception that will not fix itself, and it is the case worth stopping the lesson for.

  4. Follow the answer with “how do you know?”

    A right answer can be a guess, and a polished piece of work can be someone else’s. One follow-up question, asked in conversation, tests whether the reasoning belongs to the learner. Four kinds do most of the work: explain a choice, extend a claim, defend it against a counterexample and apply it to a case the learner has not seen. The last is as close as a short question gets to testing transfer.

04Where it breaks

A question finds only what it was built to find

A diagnostic question is limited by the misconceptions its author knew about. A new error lands in the wrong bucket or gets counted as a guess. Good wrong answers are slow to write, and a first attempt can take half an hour, which is why a shared bank built up over time matters more than any single item.

And a question that shows thinking in the moment says nothing, on its own, about whether that thinking lasts. It is evidence for a small claim: what this class believes now, and what to do next.

05Where it lands

Fewer questions, each one built

The first change is a preference for fewer, better questions. One question with three diagnostic wrong answers and a decision rule beside it tells a teacher more than ten items marked right or wrong, and it takes less time to read.

The second is a question for any item bank or quiz feature: what does a wrong answer tell the teacher? If the answer is “that the learner was wrong”, the bank was built for scoring. Scoring has its uses. It is a different job.

The third is about what a platform reports. A percentage correct is the figure least likely to change a lesson. The spread of wrong answers, grouped by the misconception each stands for, answers the question every teacher has: what do I teach again, and to whom?

The point

What transfers

The reason this page exists, rather than the story that produced it.

  1. A question shows thinking only when it is built to. Most are built to be marked.
  2. Write each wrong answer from a real misconception, so that every answer tells you something.
  3. Write the decision rule before the question is asked. A rule written afterwards bends to fit the results.
  4. Pair answers with an ungraded confidence rating. Sure and wrong is the case worth stopping for.
  5. Follow a surprising answer with “how do you know?” One follow-up tests whether the reasoning is the learner’s own.
Singapore