SET A3 PORTRAIT · PRINTS IN COLOUR
Source trailReferences

    A GENERAL SHARING

    What design is, and what designers do

    Ten questions, answered from the evidence rather than from the profession. For anybody who works alongside designers and wants to know what the job actually is.

    Three parts. What design is, what it is worth, and who does it. Eight sections, and each one is titled with the question it settles.

    Everything here is drawn from public, documented cases. No employer is named, no client work appears, and every number can be traced to a published source.

    Part one of three

    What design is

    A definition wide enough to cover the decisions that actually do the damage, and the one property that separates design that works from design that mostly does not.

    The first part builds the definition and tests it. It ends with the one property that separates design that reliably works from design that mostly does not, and with the reason the same object can work in one place and do nothing in another.

    §01

    WHAT IS DESIGN?

    AUGUST 2020 · CITIBANK

    Three tickboxes and one confirmation prompt

    A team needed to make a routine interest payment on a loan. A few million dollars. The software required three separate override boxes to be ticked to route the money correctly. They ticked one. Before the next slide: whose mistake was this?

    The system read a single ticked box as an instruction to pay off the whole loan, and released the principal to the lenders instead of holding it in an internal account.

    Almost every room answers that the mistake belongs to the person who ticked one box. That answer is available, it is intuitive, and it is the reason the same thing will happen again somewhere else this year.

    In re Citibank August 11, 2020 Wire Transfers, S.D.N.Y. 2021

    THE OUTCOME

    Citi asked for it back. A federal court said no.

    $900m

    released to Revlon’s lenders

    $0

    recovered from the disputed half

    The court applied New York’s discharge for value rule and held that lenders who received the money in good faith could keep it. More than $500 million remained in dispute and was not returned.

    Nobody in that story had designer in their job title. Somebody still decided that three unlabelled tickboxes and one confirmation prompt was an adequate way to move that sum. Nobody tested whether it was, because testing it was not anybody’s job.

    In re Citibank August 11, 2020 Wire Transfers, S.D.N.Y., February 2021

    THE DEFINITION

    Design is deciding what a system makes easy, what it makes hard, and what it makes impossible

    Every system already has those properties, whether or not anybody chose them. A form has them. So does a contract, a factory floor, a piece of legislation and an aircraft cockpit. The only question is whether somebody sat down and decided them on purpose. Mostly nobody did.

    This is a wider definition than the everyday one, and deliberately so. It covers the interface, and it also covers the contract clause, the purchasing specification and the choice of which number a team is measured on.

    The test of a definition is what it lets you see. This one makes the Citibank screen a design decision, which it plainly was, and which no narrower definition can account for.

    THE BRITISH POST OFFICE · 1999 ONWARDS

    A software fault, and the decisions sitting above it

    Post Office branches are run by subpostmasters, who are self-employed and personally responsible for the till.

    THE FAULT

    Horizon miscounted

    THE DECISIONS ABOVE IT

    Whose account of reality counts

    No screen in that story is badly drawn. What ruined those lives was a decision about whose account of reality would count, taken by lawyers and executives, none of whom would have recognised it as a design question.

    It set what was easy, what was hard and what was impossible for nine hundred people, and nobody ever examined it on those terms.

    House of Lords Library, Post Office Horizon IT scandal, 2025

    WHAT THAT DECISION COST

    Roughly nine hundred prosecutions

    ~900

    people prosecuted, 1999 to 2015

    £1.8bn

    set aside by the UK government for compensation

    Some went to prison. Many were bankrupted. Several took their own lives. Convictions were quashed in bulk by an Act of Parliament in 2024, twenty-five years after the system was introduced.

    The compensation figure is a government estimate rather than a cap, and payments across the four schemes were still running well behind it at the start of 2025.

    Post Office (Horizon System) Offences Act 2024 · House of Lords Library, 2025

    THE FIRST CONCLUSION

    Design is a category of decision, not a profession

    Designers are people who sometimes get to make those decisions and, in most organisations, do not. The decisions get made anyway, by whoever happens to be holding the pen.

    The word covers two different things that overlap much less than it suggests. One is a job, with schools, portfolios and a labour market. The other is a role, held by whoever is actually deciding what a system makes easy, hard or impossible.

    At Citibank that was almost certainly a developer implementing a written requirement. At the Post Office it was lawyers and executives. The gas fitting in the next section was settled by engineers on a standards committee.

    §02

    WHAT IS GOOD DESIGN?

    Part one of three

    §02

    What is good design?

    The most reliable finding in this whole body of evidence, and it sorts almost everything.

    Interventions that take an option away, or take effort away, hold up when they are tested properly. Interventions that add a warning, a label, a training course or an encouragement mostly shrink under scrutiny and sometimes vanish altogether.

    That single distinction does more sorting work than any framework in the field. It explains why some famous safety measures worked and others did not, why automatic pension enrolment moved a country and a tax break did not, and why so much well-intentioned design has nothing to show for itself.

    The pattern holds across medicine, aviation, public administration and personal finance, which is unusual for anything in this area. It is also the finding that reflects worst on the design profession, because most of what the profession sells sits on the losing side of the line. Empathy maps, personas, journey maps, workshops and most design systems produce information and alignment. The interventions that carry the effects produce constraints.

    TWO WAYS TO STOP THE SAME ACCIDENT

    Which one has a randomised trial behind it?

    In an operating theatre, oxygen, nitrous oxide, carbon dioxide and medical air arrive through cylinders and outlets that look broadly similar. Connect the wrong one and you can kill somebody.

    A · THE CONNECTOR

    Change the shape of the fitting

    B · THE LABEL

    Capitalise the letters that differ

    Almost every room picks the connector, because it obviously works. The connector has no trial behind it and never will have one. The lettering has a body of controlled reading studies.

    That inversion is the single most useful thing in this section, and it comes back three slides from now.

    THE DIFFERENCE

    One removes the possibility. The other improves the odds.

    Same hospital, same accident, same profession designing the fix. Only one of them works whether or not anybody is paying attention.

    FIG 01 — ONE FIX STOPS THE ERROR. THE OTHER ONLY MAKES IT NOTICEABLE.

    The gap between removing a possibility and improving a chance is what separates design that reliably works from design that mostly does not.

    It is also why the connector needs no evidence. A fitting that does not fit cannot be attached to the wrong cylinder, and that is a matter of logic rather than statistics.

    THE PATTERN, STATED

    Take an option away and it holds up. Ask people to be careful and it fades.

    This holds across medicine, aviation, public administration and personal finance, which is unusual for anything in this field.

    The evidence for it is unusually clean, and the next four slides are the two halves of it. First the interventions that remove something, then the ones that add information.

    Nothing here depends on any particular design method. It is a property of the intervention, not of the process that produced it.

    REMOVING THE EFFORT · PENSIONS

    The same scheme, the same money, the same people

    American workplace pension schemes used to require employees to sign up. Researchers studied one large firm that switched to enrolling people automatically, with the option to leave.

    FIG 02 — JOINING BECAME THE DEFAULT AND PARTICIPATION MORE THAN DOUBLED

    Most employees never changed the contribution rate or the fund they had been placed in, which is the finding underneath the headline. The default did not persuade anybody. It simply became what happened.

    A much larger study using the Danish government’s records for the entire saving population found that roughly 85% of people never actively decide anything about their pension. Automatic employer contributions passed through to total wealth almost krone for krone. Cutting the tax break on saving, which is the standard economic lever, moved almost nobody.

    Madrian & Shea 2001, QJE · Chetty et al. 2014, QJE

    REMOVING THE OPTION · AVIATION, 1947

    Four hundred and sixty crashes filed as pilot error

    Two research psychologists went through the reports for the US Army Air Forces and found pilots pulling up the landing gear instead of the flaps on landing, because the two levers sat next to each other and felt identical.

    Somebody shape-coded them. A small wheel for the landing gear, a wedge for the flaps, so a gloved hand could tell them apart without looking. The convention went into the standards and is still there.

    Those crashes had been recorded as pilot error for years. The pilots were never the problem, and this is the single most transferable move in the whole field. It comes back in section seven.

    Fitts & Jones 1947, Aero Medical Laboratory

    ADDING INFORMATION · THE OTHER HALF

    What happens when you tell people instead

    A pulse oximeter is the clip that goes on a fingertip and reports blood oxygen. It is the most recognisable piece of anaesthetic safety equipment in the world.

    20,802

    patients randomised, with a pulse oximeter or without

    0

    difference in deaths or serious complications

    49–96%

    of prescribing alerts overridden by clinicians

    FIG 03 — THE OVERRIDE RATE ACROSS THE PUBLISHED STUDIES

    The alert figure is the more damning of the two. Hospital prescribing systems produce a warning when two drugs might interact badly. The warnings are frequent enough and irrelevant enough that clicking past them becomes automatic, which means a safety feature has trained its users to ignore it.

    Tall-man lettering sits in the same category. It performs well in controlled reading tests and has thin, mixed evidence on whether it prevents real dispensing errors.

    Møller et al. 1993, Anesthesiology · van der Sijs et al. 2006, JAMIA

    AND THE HONEST SIZE OF ALL OF IT

    Six times larger in the journals than in the field

    Two government teams, one American and one British, spent years running proper randomised experiments on redesigned letters, forms, reminders and defaults. 126 trials, more than 23 million people.

    FIG 04 — IF 40 IN 100 WERE DOING IT BEFORE, ABOUT 41 DO IT AFTER

    The gap is almost entirely explained by publication bias and statistical power. Journals publish the experiments that worked, and small experiments produce wilder numbers in both directions. The academic trials had a median sample of a few hundred. The government trials averaged ten thousand.

    Anybody making a business case from published behavioural research is quoting a number roughly six times too large. This is the honest figure.

    DellaVigna & Linos 2022, Econometrica

    THE ANSWER TO “WHAT IS GOOD DESIGN”

    Good design removes an option. Weak design asks people to be careful.

    It is not about taste, and it does not need a bigger budget. It needs somebody to ask what this makes impossible rather than what it makes clearer.

    This is the line worth leaving with. Everything in the rest of the talk either follows from it or tests it.

    It also gives you something to do in a meeting. When somebody proposes a fix, ask whether it removes the possibility of the failure or merely improves the chance that somebody spots it. The second kind is not worthless, but it should never be the only thing on the table.

    §03

    WHY DOESN’T GOOD DESIGN TRAVEL?

    Part one of three

    §03

    Why doesn’t good design travel?

    The same object, used in two places, produced opposite results.

    Almost everything a design team produces is an artefact. A screen, a form, a document, a checklist. It is natural to assume the artefact is the thing that works, because the artefact is the thing you can hold.

    The surgical checklist is the clearest evidence that this assumption is wrong. It is one page, it costs nothing, and it halved deaths in the study that made it famous. Mandated across a much larger population, it did nothing measurable at all.

    The standard reading, which the surrounding literature supports, is that the checklist was never the intervention. What worked in the original sites was the briefing, the verification behaviour, and the fact that a junior nurse now had explicit permission to stop a senior surgeon. The Dutch SURPASS study, which redesigned the whole perioperative pathway rather than issuing a form, found effects at least as large as the original.

    THE SAME ONE-PAGE CHECKLIST, TWICE

    Predict the second result before you see it

    Nothing exotic on it. Confirm the patient and the operation. Confirm which side of the body. Confirm the antibiotics were given. Everyone in the room says their name and role out loud.

    A · EIGHT HOSPITALS, 2009

    The study that made it famous

    B · ONTARIO, 2014

    The same checklist, made compulsory

    Almost everybody predicts a smaller positive effect, on the reasoning that a mandate dilutes an intervention without destroying it. That is a sensible prediction and it is wrong.

    Committing to an answer before the reveal is worth doing properly. A wrong prediction here teaches more than a right one, and the reason it is wrong is the point of the section.

    Haynes et al. 2009, NEJM · Urbach et al. 2014, NEJM

    THE ANSWER

    No reduction in deaths. No reduction in complications. Nothing at all.

    Same checklist. Twenty-five times the sample. The larger and better-identified study found no measurable effect.

    FIG 05 — THE SAME PIECE OF PAPER, AND OPPOSITE RESULTS

    Ontario used an interrupted time-series design across the province, which is a stronger way of establishing causation than the before-and-after comparison in the original study. The null result is not a weaker finding. It is a better-identified one.

    A Dutch study that redesigned the whole perioperative pathway, rather than issuing a form, found effects at least as large as the 2009 result. Complications fell from 27.3 to 16.7 per 100 patients.

    Urbach et al. 2014, NEJM · de Vries et al. 2010, NEJM

    WHAT ACTUALLY CARRIED THE EFFECT

    The paper was the occasion, not the intervention

    What worked in the first eight hospitals was the briefing, the checking behaviour, and the fact that a junior nurse now had explicit permission to stop a senior surgeon.

    Ontario made the paper compulsory and audited the signatures, and got signed pieces of paper. The behaviour that produced the original result was never the thing being copied, because nobody had identified it as the thing that worked.

    This is the most common failure in organisational change, and it is not confined to hospitals. Copy the artefact, audit the compliance, wonder why nothing moved.

    THE INVERSION

    Nobody has ever run a trial on the gas fitting

    There is no study, and there never will be, because a connector that does not fit cannot be attached to the wrong cylinder. The proof is logical rather than statistical. You do not need evidence for something that cannot happen.

    Which produces a real problem for anybody trying to evaluate design by looking at the research. The cheap things to test are the low-consequence things, so the strength of the evidence runs roughly opposite to the size of the effect.

    Anybody who judges design by asking where the study is will reliably fund the wrong work. Anybody who argues for design using studies will reliably argue for the wrong work.

    WHAT TO TAKE FROM THIS SECTION

    Rank these interventions by weight of evidence and you get roughly the reverse of ranking them by value

    The best-evidenced work is rarely the most valuable work. That is a property of what is cheap to measure, not a property of what matters.

    It also explains something that puzzles a lot of executives. The design work that produces the cleanest before-and-after chart is usually the work that mattered least, because a clean chart requires a measurable, repeatable, low-stakes change.

    Section six is about what to do with that, because it is a real problem rather than an excuse.

    Part two of three

    What it is worth

    What bad design costs, what the profession’s own value numbers are worth, and where in an organisation this work pays.

    The second part is about money and consequence. It starts with the failure record, because that is where the largest measured effects are, and only then turns to the business-value numbers the profession usually leads with.

    §04

    WHAT DOES BAD DESIGN COST?

    Part two of three

    §04

    What does bad design cost?

    The largest measured effects of design on the world are its failures.

    The positive research in this area deals in single-digit percentage points and contested effect sizes. The failure record deals in hundreds of deaths, hundreds of wrongful convictions and sums in the billions.

    None of the cases in this section was produced by anybody with design in their job title. Every one of them was a design decision, in the sense this talk uses the word.

    Which means the honest inference is not that these organisations should have hired more designers. It is that a class of decision with enormous consequence is currently made without method, review or accountability, by people who do not know they are making it.

    BOEING 737 MAX · 2018 AND 2019

    Two crashes, twenty months apart

    346

    people killed across the two flights

    $18bn+

    cost, by Boeing’s own January 2020 statement

    Boeing added software that pushed the aircraft’s nose down whenever it believed the plane was about to stall. The software judged that by reading a single sensor on the outside of the fuselage, with no second sensor to check it against.

    When that sensor failed, the software pushed the nose down again, and again. Boeing later paid $2.5 billion to settle a US fraud conspiracy charge over what it had told regulators about the system.

    NTSB findings 2019 · KNKT final report 2019 · DOJ settlement, January 2021

    THE ARCHITECTURE

    One sensor, no cross-check, and a four-second assumption

    The aircraft carried two angle-of-attack sensors. The system read one of them. The safety case assumed the crew would diagnose an uncommanded activation and correct it in about four seconds, tested in isolation rather than against the real cockpit.

    FIG 06 — THE SECOND SENSOR EXISTED AND WAS NEVER CONSULTED

    Investigators found Boeing had assessed pilot response to an uncommanded activation on its own, not against the environment the accident crews actually faced, which included a stick shaker and simultaneous airspeed and altitude disagreement warnings.

    Post-accident simulator work suggested crews had roughly forty seconds to diagnose and execute the runaway-trim procedure once the full alert cascade began. Four seconds was a design assumption written as an engineering parameter.

    NTSB safety recommendations, September 2019

    AND THE SAME SHAPE, ELSEWHERE

    Four more, none of them designed by a designer

    01Knight Capital, August 2012 A deployment left obsolete test code active on one of eight production servers. It generated runaway orders across 154 stocks in forty-five minutes, roughly four million trades. The SEC order found systemic governance failure rather than a single bug: no adequate real-time monitoring and no kill switch. Knight lost over $460 million and was sold within days.
    02The butterfly ballot, November 2000 Peer-reviewed statistical analysis found Buchanan’s vote in Palm Beach County was several thousand higher than any model of the surrounding counties predicts. Nobody can prove how those voters would otherwise have voted. The number of votes the layout appears to have misdirected was several times larger than the 537-vote margin that decided the presidency.
    03The Hawaii false missile alert, 13 January 2018 The FCC investigation found the alert-origination software listed the internal test alert and the live missile alert in the same dropdown, with an identical, content-free confirmation prompt for both. No colour, no watermark, no separate environment. The alert went to every phone in the state and took thirty-eight minutes to retract.
    04The NHS National Programme for IT, 2002 to 2011 The National Audit Office estimated total cost at £12.7 billion in 2008, later revised to £9.6 billion. By August 2011, £7.3 billion had been spent when the Public Accounts Committee recommended scrapping it. The core patient-record elements were never delivered as originally specified.

    Each of these is filed somewhere as an IT failure, a governance failure or a human error. Each is a decision about what a system made easy, hard or impossible, taken by somebody who would not have described it that way.

    The Hawaii case is the purest. Somebody put a live state-wide emergency broadcast and a test message in the same list, with the same confirmation prompt.

    SEC Release 70694 · Wand et al. 2001, APSR · FCC report 2018 · NAO and PAC reports

    AND IT WORKS THE OTHER WAY TOO

    Two clicks to join. Four pages, six clicks and fifteen options to leave.

    In 2025 the US consumer protection regulator settled with Amazon for $2.5 billion over the way you cancel a Prime subscription. Amazon had measured internally how much revenue each additional obstacle produced.

    The cancellation flow was named internally after the Iliad, for its length. Every step offered a reason to stay. The FTC complaint describes a four-page, six-click, fifteen-option sequence against a two-click sign-up.

    Independent research found deceptive interface patterns on about 11% of 11,000 shopping sites, and 22 third-party vendors selling the functionality as an off-the-shelf service. This is a supply chain, not isolated bad judgement.

    FTC v. Amazon.com, September 2025 · Mathur et al. 2019, ACM CSCW

    THE UNCOMFORTABLE PART

    The skills were not the problem. The objective was.

    A clear goal, careful research, honest measurement and iteration on the results. All of it in the service of stopping customers from doing something they had already decided to do.

    Any argument that good design creates value has to survive this observation. The field’s most rigorously measured commercial successes include its abuses, and they were produced by the same skills applied with more discipline than most well-intentioned work receives.

    Which is a reason to care who holds these decisions and what they are held to, rather than a reason to trust the profession by default.

    §05

    IS DESIGN VALUABLE?

    Part two of three

    §05

    Is design valuable?

    Before going any further, the numbers my own side usually quotes.

    Having made the case, it is worth being straight about the evidence that gets used to make it. Most of the business-value numbers in circulation do not survive a check, and anybody in this room who has seen them deserves to know that.

    This is placed here rather than at the start on purpose. The accident record has already established that these decisions matter. What follows is a separate question, which is whether anybody has ever demonstrated a financial return on employing a design team.

    One general rule falls out of all of it. Every number the design profession quotes about its own value was produced by somebody with a commercial interest in the answer, using a method they did not publish, on a sample they selected. The one exception, an EU study of more than 12,000 firms, reports a null.

    THE NUMBERS MY OWN SIDE QUOTES

    Three you have probably been shown

    FIG 07 — THE INDEX RESTS ON ONE OF ITS SIXTEEN COMPANIES

    01The design value index Sixteen companies, selected by an organisation whose stated mission is to advocate for design’s economic importance. No published selection criteria, no sector matching, no risk adjustment, and no confidence interval on a portfolio of sixteen. Apple was one of them, and removing Apple collapses the result. The index has not been updated since around 2017.
    02The consultancy design index Around 300 firms scored on twelve proprietary criteria, built largely from structured interviews with those firms’ own design leadership, then correlated with five-year revenue growth. No company list, no rubric weights, no coefficients, no standard errors, no significance tests, and no replication in the eight years since. The input variable was supplied by the party being assessed.
    03The four-to-one return This does not come from the national business survey it is usually attributed to. Tracing it through the Design Council’s own 2013 parliamentary submission, it originates in the evaluation of Designing Demand, a subsidised design-support programme for small firms running from 2007 to 2012. Firms self-selected into applying and reported outcomes to an evaluator commissioned to assess value for money.

    I have quoted two of these in a presentation. Nobody ever asked me where they came from, which is how a number survives for a decade without a source. Everybody in the room wants it to be true, so nobody checks.

    None of this concedes the argument. The accident record in the last section stands on its own. What it does is clear the ground.

    DMI Design Value Index · McKinsey 2018 · Design Council parliamentary submission 2013

    THE ONE STUDY DONE PROPERLY

    More likely to innovate. No detectable effect on the money.

    The most careful piece of research in this area was done at the European Commission’s own research centre. More than 12,000 firms across Europe, with proper statistical handling of company size, age, industry and country.

    Firms that invest in design turn out to be meaningfully more likely to bring something new to market. There is no significant effect on the turnover they earn from doing so, once the way design is embedded in the organisation is controlled for.

    That is the strongest evidence the field has, and it does not support the claim the field most wants to make. The authors also test explicitly for selection bias, which almost nothing else in this literature does.

    Montresor & Vezzani 2017, EU Joint Research Centre

    FROM SOMEBODY WHO SOLD IT

    Nussbaum did more than almost anybody to popularise design thinking as a magazine editor, and then called it a failed experiment.

    Companies absorbed the process of Design Thinking all too well, turning it into a linear, gated, by-the-book methodology that delivered, at best, incremental change.

    Bruce Nussbaum, Fast Company, 2011

    The branded five-stage method taught in business schools is a different thing from the definition this talk uses, and it is worth knowing how thin its evidence is. In the entire research literature there is one properly randomised trial of it.

    Three hours of teaching, over three days, with Indian middle-school students. Idea fluency went up. Confidence went up, especially among the girls. Originality and flexibility went down.

    Nussbaum 2011, Fast Company · Rao, Puranam & Singh 2022, Innovation

    THE HONEST POSITION

    Two different claims, run together for a generation

    The consequences of design decisions are enormous and thoroughly documented. The financial return on employing a design team is neither, and no credible study has established it. Trying to prove the second using the first is how the profession ended up with a stock index of sixteen companies.

    This distinction matters for anybody deciding whether to fund a design function. The case for caring about these decisions is overwhelming. The case for a particular headcount is a separate argument that has to be made on its own terms, situation by situation.

    Section six is the tool for making it, because it tells you which decisions in your own organisation are worth the attention.

    §06

    WHERE IS DESIGN WORTH MOST?

    Part two of three

    §06

    Where is design worth most, and where not at all?

    Two things decide what a design decision is worth, and they are independent of each other.

    Consequence is whether the decision sits on the path by which the system actually fails, and whether it repeats. Credit is whether anybody can see the effect, attribute it and count it.

    Those two vary independently, which is the whole problem. The work with the most consequence is frequently the work with the least visible result, and the work that produces the most evidence is frequently the work that matters least.

    This is also the most useful thing in the talk for anybody commissioning design rather than doing it. It tells you where to point a design team, and it tells you why the evidence they bring you looks so thin next to the claims they make.

    TWO VARIABLES

    Consequence and credit vary independently

    Consequence is whether the decision sits on the path by which the system actually fails, and whether it repeats. Credit is whether the effect can be seen, attributed and counted, and whether it shows up before the next performance review.

    Infection rates are recorded whether or not anybody is thinking about design, so an improvement lands on somebody’s chart within months. An accident that never happened is recorded nowhere, by anyone, ever.

    Those two facts are what make design hard to fund, hard to evaluate and hard to hire for, and they are the same two facts.

    FOUR SITUATIONS

    All the evidence comes from one column

    Put consequence and credit together and you get four boxes. Almost every piece of evidence design offers about itself comes from the right-hand column, because those are the only two boxes that produce evidence at all.

    FIG 08 — THE VALUE IS ON THE LEFT. THE EVIDENCE IS ALL ON THE RIGHT.

    A connector that cannot be misconnected generates no data. An accident that does not happen generates no report. Meanwhile a visual refresh that lifts sales half a percent generates a chart within a fortnight.

    So the profession has arranged itself around the work it can count, and then objects when executives treat design as the function that produces small numbers. The executives are reading the evidence the profession handed them.

    WHERE THE LARGEST VALUE SITS

    Nobody has ever been promoted for a disaster that failed to occur

    Los Angeles wrote a schools technology purchasing specification that a federal review found had effectively excluded every supplier that was not Apple, then spent about $1.3 billion acting on it. One person rewriting that document could have changed the outcome of the whole programme, and nobody would have been able to see that they had done anything.

    The iPads arrived preloaded with a curriculum that two of the sixty-nine trial schools ended up using. The specification was written before anybody with a design title was involved, and it fixed the outcome more completely than any subsequent decision could.

    Purchasing documents, standards, defaults and interlocks are the highest-multiplier objects in most large organisations. Deciding once, for everybody, is what a headquarters is for.

    AND WHERE IT WILL NOT PAY

    Six situations worth checking your own against

    01Buyer and user are different people Test: can you name the budget holder, and do they personally use the product weekly? If not, the return on experience quality is structurally capped, because the evaluation criteria are fixed independently of daily use.
    02Switching costs exceed the achievable improvement Test: price the cost of leaving in money and time against the best plausible gain from a redesign. Where the first dominates, better design cannot move market share, which is why several widely disliked systems have kept their customers for decades.
    03Network effects dominate Test: does the value to a new user scale mainly with how many others are already present? If so, design cannot substitute for missing scale. This is the mechanism behind marketplaces, file formats and messaging networks alike.
    04Price or allocation is administered Test: is price or access set by a regulator, a sole-source contract or a fixed reimbursement schedule rather than by customer choice among competing offers? If so, the channel from experience to revenue is absent by construction.
    05The attribute is saturated Test: can a competent generalist hit the category baseline using off-the-shelf components? Open component libraries have made a clean, consistent, accessible interface available for free, which moved a large share of what design used to differentiate into the category of hygiene.
    06Design is engaged downstream of value commitment Test: does the design function influence price, unit cost or architecture, or only the surface built on decisions made elsewhere? Expected return is capped by the surface’s share of the total value at stake.

    None of these is a statement about an industry. Each describes where a particular decision sits in relation to the way the product actually succeeds or fails, and the same company will contain decisions that fail every test and decisions that pass all of them.

    Which is why asking whether a sector values design is the wrong question. The useful question is how close a given decision sits to the way that thing actually fails.

    THE EXCEPTION THAT OVERRIDES ALL SIX

    Wherever the interaction is the mechanism, design comes straight back

    Pharmaceuticals fails several of those tests and asthma inhalers are still worth lives, because a drug delivered through a device most patients use incorrectly is a weaker drug.

    The US Food and Drug Administration publishes human factors guidance for combination products precisely because inhaler and auto-injector usability errors cause clinical harm. Persistent inhaler technique errors remain a documented barrier to effective treatment after decades of device iteration.

    The test is not what industry you are in. It is whether the interaction sits on the causal path between what a person does and what the product delivers.

    FDA human factors guidance for combination products, 2016

    Part three of three

    Who does it

    How the job differs from an analyst’s or an engineer’s, what separates a good practitioner, and what you would ask one.

    The last part is about people. It separates this job from the ones next to it, and then asks what a good practitioner actually does differently and how anybody outside the field would recognise one.

    §07

    WHO IS A DESIGNER?

    Part three of three

    §07

    Who is a designer, and how is the job different?

    Analyst, engineer, developer, product manager, designer. Each owns a different question.

    The unhelpful version of this distinction says that designers care about users and other roles do not. That is both flattering and false. Engineers care about users. Analysts care about users. Nobody sets out to build the wrong thing.

    The useful version is that these roles are distinguished by the question each one owns, rather than by what each one produces. Every question on that list needs asking. The failure mode is when one of them belongs to nobody.

    A designer with no engineer produces something that does not work. An engineer with no analyst produces something nobody can tell is working. An analyst with no designer optimises a number that turns out to be the wrong number. These are complements rather than competitors, and treating them as competitors is how organisations end up with all four roles filled and the impossible-question still unowned.

    THE SENTENCE THAT SORTS THE ROLES

    MCAS worked exactly as specified

    Every line of that software did what it was written to do. It read its sensor, judged the aircraft to be stalling and moved the tail. The engineering was not defective. The specification was.

    This is the cleanest way to see what separates these jobs. An engineer asks whether the thing works as specified, and answers yes. That answer is correct and it is not the whole question.

    Somebody has to ask whether one sensor was enough, and what happens to a crew who meet this for the first time at four hundred feet with three other alarms going. That question belongs to somebody, and on the 737 MAX programme it appears to have belonged to nobody in particular.

    FIVE ROLES, FIVE QUESTIONS

    Distinguished by the question they own, not by what they produce

    The unhelpful version of this says designers care about users and other people do not. That is flattering and false. Engineers care about users. Analysts care about users. Nobody sets out to build the wrong thing.

    FIG 09 — EVERY QUESTION NEEDS ASKING. THE LAST ONE OFTEN BELONGS TO NOBODY.

    These are complements rather than competitors. A designer with no engineer produces something that does not work. An engineer with no analyst produces something nobody can tell is working. An analyst with no designer optimises a number that turns out to be the wrong number.

    Treating them as competitors is how organisations end up with all five roles filled and the last question still unowned.

    THE SAME PROJECT, TWO CORRECT ANSWERS

    Organ donation, read by an analyst and by a designer

    Switching a country from opt-in to opt-out moves registered consent from somewhere between 4% and 27% up to between 86% and almost 100%. The two distributions do not overlap.

    THE ANALYST IS RIGHT

    The number moved enormously

    THE DESIGNER’S QUESTION

    Was consent the right number?

    Neither column contains a mistake. The analyst measured the thing the policy was defined against. The design question is whether that definition was right in the first place, and that question is not answerable from inside the measurement.

    Wales, ten years after introducing opt-out, reports that the anticipated increase in consent has not translated into the outcome it was introduced for.

    Johnson & Goldstein 2003, Science · Arshad et al. 2019, Kidney International

    THE FAILURE MODE

    A question that belongs to nobody gets answered by accident

    At Citibank the developer implemented the requirement, the requirement was signed off, the payment ran, and nobody in that chain was ever asked whether three tickboxes were an adequate control on nine figures.

    Every role in that story did its job. The failure was not in any of the answers. It was that one of the questions had no owner, so the answer arrived by default.

    Which is a more useful thing to look for in your own organisation than a headcount gap. Take the last significant thing that went wrong and ask which of the five questions nobody was asking.

    THE PRACTICAL VERSION

    On a good team it barely matters who asks it, only that somebody does

    Plenty of engineers ask the impossible question. Plenty of analysts do. The job title is a poor guide, which is the whole argument of this talk turned round.

    Hiring a designer is one way to make sure the question gets asked. It is not the only way, and it does not work on its own. Plenty of organisations have a design team and still never ask it, because the team is engaged after the specification is locked.

    The reliable version is to name the question, attach it to a decision point, and make somebody answer it in writing before the money is committed.

    §08

    WHAT DOES A GOOD ONE DO?

    Part three of three

    §08

    What does a good one do, and how would you know?

    This last part is about the people, and about what you would actually ask them.

    Everything so far has been about the subject. This section is about the person, because most people in this room will at some point hire, brief or sit next to somebody doing this work, and the usual ways of assessing them do not work.

    The short version is that a portfolio shows you what somebody was permitted to work on. It shows you the version that shipped. It does not show you the decision, the alternative, or what happened afterwards.

    THE USUAL WAY OF ASSESSING SOMEBODY

    A portfolio shows you what somebody was permitted to work on

    Twenty screens, a case study, a before and after. It shows the version that shipped. It does not show the decision, the alternative, or what happened to any of it afterwards.

    If the artefact is not the intervention, which section three established, then a folder of artefacts is close to the least informative thing you could ask somebody for.

    It also selects for work that was visible and pleasant to look at, which is the low-consequence, high-credit corner of the grid from section six. The best work in this field photographs badly.

    WHAT A GOOD ONE DOES BETTER

    They can tell apart two things that look the same

    This is more specific than taste, and it is trainable. Every pair here is indistinguishable from any distance, and each one came out differently.

    FIG 10 — JUDGEMENT IS SEEING THE DIFFERENCE BEFORE THE RESULT COMES IN

    Research on interleaved practice contains a specific finding most summaries miss. Mixing categories helps a great deal when they are easy to confuse and helps not at all when they are not. Paintings by different artists show a large benefit. Foreign vocabulary shows none. The benefit comes from the discrimination rather than the variety.

    Which means a library of good examples builds judgement slowly, and a library of near-identical pairs with opposite outcomes builds it fast. This talk has been assembling one throughout.

    Brunmair & Richter 2019, Psychological Bulletin

    AND THEY KNOW HOW OFTEN THEY ARE WRONG

    In the one place that keeps score

    Large web companies test changes by showing one version to half their users and another to the other half. It is about as close to a controlled experiment as commercial life gets.

    ~33%

    of tested ideas at Microsoft improve what they were built to improve

    59%

    accuracy predicting the winner, against a 50% coin flip

    FIG 11 — BETTER THAN CHANCE, AND NOT BY MUCH

    Amazon’s reported success rate is below half. The single best revenue idea in Bing’s history, a change to how advertising headlines were displayed, was worth more than $100 million a year and sat in a backlog for six months marked low priority, because nobody who looked at it could tell it was worth anything.

    A good designer treats that as the baseline condition of the work and builds a process that survives being wrong. Small releases, reversible decisions, a cheap way to find out early. A weak one argues from conviction and finds out at scale.

    Kohavi, Crook & Longbotham 2009 · Kohavi & Thomke 2017, HBR

    AND THEY CARRY NEWS NOBODY WANTS

    Somebody has to say the new system is worse than the old one

    Los Angeles built a student records system that engineers load-tested against 1,200 users for a district serving hundreds of thousands. Nearly every school reported it could not reliably identify students with disabilities. Leadership ran a scheme called feed the pig, fining staff a quarter every time somebody said the old system had been better.

    A team that charges people for reporting a fault has built a machine for destroying the only signal capable of saving the project. Somebody in that building needed to say so, out loud, to people who had just made it expensive.

    Doing that from a position of low formal authority, repeatedly, without becoming tiresome, is a real and rare capability. It is also the part of the job that nobody puts in a portfolio.

    HOW SOMEBODY GETS GOOD AT ANY OF THIS

    The feeling of getting better runs opposite to getting better

    Researchers randomly assigned Harvard undergraduates to active or passive versions of identical physics material, with trained instructors on both sides. The active students learned more. They reported learning less, and rated the lecturer who taught them least as the better teacher.

    FIG 12 — THEY LEARNED MORE AND FELT THEY HAD LEARNED LESS

    The gap is about half a standard deviation in each direction, which in a class is roughly the difference between a middling student and one in the top third. The feeling moved as far in the wrong direction as the learning moved in the right one.

    Which is why conference talks, newsletters and case studies feel productive and mostly are not. Reading a case and thinking that makes sense is the exact sensation those students had while learning less. What works is reconstructing a case from memory a week later, keeping pairs rather than examples, and writing your prediction down before you look.

    Deslauriers et al. 2019, PNAS

    SO HOW WOULD YOU KNOW

    Six measures that do not work, and six questions that do

    WHAT IS USUALLY USED

    And why each one fails

    WHAT WOULD WORK

    None of it survives a portfolio review

    All six of the questions on the right are retrospective, and most of the good answers describe non-events. None of them survives a forty-five minute interview, because each requires the candidate to have kept records that nobody ever asked them to keep.

    So hiring keeps selecting for the artefact, which is the one thing that does not carry the effect. Portfolios get better. Judgement gets no easier to see.

    Nielsen Norman Group, survey of 557 practitioners across 356 companies

    THE ONE QUESTION

    What did you make impossible?

    A strong answer names a specific failure that used to occur, describes the change that removed the possibility of it, and can say how anybody knows. A weak answer describes something that looks better than it did before.

    It takes about a minute to ask and, unlike a portfolio, it cannot be assembled the night before. It works on a candidate, on a supplier, on a team and on your own last twelve months.

    The person and the field have exactly the same problem underneath all of this. Design’s value is hard to prove because its successes are things that did not happen, and a designer’s value is hard to prove for precisely the same reason.

    Somebody in your organisation is drawing the next three-tickbox screen this week.

    They will file it as a requirement, which is how these decisions almost always arrive.

    Nobody will check what it makes impossible until it costs something.

    THANK YOU

    Thank you

    The two long-form articles behind this talk go further on the evidence, the caveats and the sources. Happy to send them, and happy to argue about any of it.

    §08 · WHAT A GOOD ONE DOES

    Source trail

    References

    1. Arshad, A., Anderson, B., & Sharif, A. (2019). Comparison of organ donation and transplantation rates between opt-out and opt-in systems. Kidney International, 95(6), 1453–1460. https://doi.org/10.1016/j.kint.2019.01.036
    2. Bhargava, S., & Manoli, D. (2015). Psychological frictions and the incomplete take-up of social benefits: Evidence from an IRS field experiment. American Economic Review, 105(11), 3489–3529. https://doi.org/10.1257/aer.20121493
    3. Brunmair, M., & Richter, T. (2019). Similarity matters: A meta-analysis of interleaved learning and its moderators. Psychological Bulletin, 145(12), 1029–1052. https://doi.org/10.1037/bul0000209
    4. Chetty, R., Friedman, J. N., Leth-Petersen, S., Nielsen, T. H., & Olsen, T. (2014). Active vs. passive decisions and crowd-out in retirement savings accounts. Quarterly Journal of Economics, 129(3), 1141–1219. https://doi.org/10.1093/qje/qju013
    5. DellaVigna, S., & Linos, E. (2022). RCTs to scale: Comprehensive evidence from two nudge units. Econometrica, 90(1), 81–116. https://doi.org/10.3982/ECTA18709
    6. Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. PNAS, 116(39), 19251–19257. https://doi.org/10.1073/pnas.1821936116
    7. de Vries, E. N., Prins, H. A., Crolla, R. M. P. H., et al. (2010). Effect of a comprehensive surgical safety system on patient outcomes. New England Journal of Medicine, 363(20), 1928–1937. https://doi.org/10.1056/NEJMsa0911535
    8. Federal Communications Commission. (2018). Report on the false emergency alert issued in Hawaii on January 13, 2018. https://www.fcc.gov/document/fcc-releases-report-hawaii-false-emergency-alert
    9. Federal Trade Commission. (2025). FTC v. Amazon.com, Inc. Settlement announcement. https://www.ftc.gov/news-events/news/press-releases
    10. Fitts, P. M., & Jones, R. E. (1947). Analysis of factors contributing to 460 “pilot-error” experiences in operating aircraft controls. Aero Medical Laboratory, Air Materiel Command.
    11. Haynes, A. B., Weiser, T. G., Berry, W. R., et al. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine, 360(5), 491–499. https://doi.org/10.1056/NEJMsa0810119
    12. House of Lords Library. (2025). Post Office Horizon IT scandal. UK Parliament. https://lordslibrary.parliament.uk/
    13. Johnson, E. J., & Goldstein, D. (2003). Do defaults save lives? Science, 302(5649), 1338–1339. https://doi.org/10.1126/science.1091721
    14. Kohavi, R., Crook, T., & Longbotham, R. (2009). Online experimentation at Microsoft. Third Workshop on Data Mining Case Studies, KDD 2009.
    15. Kohavi, R., & Thomke, S. (2017). The surprising power of online experiments. Harvard Business Review, 95(5), 74–82. https://hbr.org/2017/09/the-surprising-power-of-online-experiments
    16. Linowski, J. (2022). Beyond opinions about opinions: What 70,149 guesses tell us about predicting A/B tests. GoodUI. https://goodui.org/blog/
    17. Madrian, B. C., & Shea, D. F. (2001). The power of suggestion: Inertia in 401(k) participation and savings behavior. Quarterly Journal of Economics, 116(4), 1149–1187. https://doi.org/10.1162/003355301753265543
    18. Mathur, A., Acar, G., Friedman, M. J., et al. (2019). Dark patterns at scale. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), Article 81. https://doi.org/10.1145/3359183
    19. McKinsey & Company. (2018). The business value of design. McKinsey Quarterly.
    20. Mertens, S., Herberz, M., Hahnel, U. J. J., & Brosch, T. (2022). The effectiveness of nudging: A meta-analysis of choice architecture interventions. PNAS, 119(1). https://doi.org/10.1073/pnas.2107346118
    21. Møller, J. T., Johannessen, N. W., Espersen, K., et al. (1993). Randomized evaluation of pulse oximetry in 20,802 patients: II. Anesthesiology, 78(3), 445–453.
    22. Montresor, S., & Vezzani, A. (2017). Design, innovation and performance in European firms (JRC Working Paper 01/2017). European Commission Joint Research Centre.
    23. National Transportation Safety Board. (2019). Safety recommendations to the Federal Aviation Administration on the Boeing 737 MAX. https://www.ntsb.gov/
    24. Nussbaum, B. (2011, April 5). Design thinking is a failed experiment. So what’s next? Fast Company. https://www.fastcompany.com/1663558/
    25. Pronovost, P., Needham, D., Berenholtz, S., et al. (2006). An intervention to decrease catheter-related bloodstream infections in the ICU. New England Journal of Medicine, 355(26), 2725–2732. https://doi.org/10.1056/NEJMoa061115
    26. Rao, H., Puranam, P., & Singh, J. (2022). Does design thinking training increase creativity? Innovation: Organization and Management, 24(2), 315–332. https://doi.org/10.1080/14479338.2021.1897645
    27. Securities and Exchange Commission. (2013). In the Matter of Knight Capital Americas LLC (Release No. 70694). https://www.sec.gov/litigation/admin/2013/34-70694.pdf
    28. Urbach, D. R., Govindarajan, A., Saskin, R., Wilton, A. S., & Baxter, N. N. (2014). Introduction of surgical safety checklists in Ontario, Canada. New England Journal of Medicine, 370(11), 1029–1038. https://doi.org/10.1056/NEJMsa1308261
    29. van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006). Overriding of drug safety alerts in computerized physician order entry. JAMIA, 13(2), 138–147. https://doi.org/10.1197/jamia.M1809
    30. Wand, J. N., Shotts, K. W., Sekhon, J. S., et al. (2001). The butterfly did it: The aberrant vote for Buchanan in Palm Beach County, Florida. American Political Science Review, 95(4), 793–810. https://doi.org/10.1017/S000305540040002X

    End of the talk

    Back to geraldajam.com