What design is, and how it gets practised in edtech

The situation

Educational technology has been studied for forty years, and the headline result is that it does not work. Tamim and colleagues pooled twenty-five meta-analyses covering more than a thousand primary studies and found an average effect of about a third of a standard deviation, with heterogeneity high enough to make the average close to meaningless. Bernard and colleagues found the same thing from the other direction: comparing one medium against another produces an effect near zero, and comparing one design of interaction against another produces something worth having.

So the average is the least interesting number in the field. The spread around it is the phenomenon, and a serious share of the spread is design — not the look of the thing, but what it asks a learner to do. Which makes it strange that in most edtech work design is admitted at the interaction layer and excluded from the decision layer. Someone will ask for a screen to be improved. Far fewer people ask what a learner will actually do inside it, whether the thing needs to exist, or what would have to be true for it to work. Those are the questions the variance is made of.

I have been the designer handed a finished decision and asked to make it good, and I took too long to notice that this was not the job. At the time it felt like being trusted.

The question

What if design is not a layer applied to a decision, but a category of decision in its own right — the decisions that set what a system makes easy, hard, and impossible?

Define it that way and the population changes completely. Most decisions in the category are made by people who do not hold the job title and would not use the word: a developer implementing a written requirement, a lawyer drafting a policy, somebody writing a procurement specification. Design as a profession is a job, with schools and portfolios and a labour market. Design as a category is a role, held by whoever is actually deciding what the system will permit. The two overlap much less than the single word suggests, and most of the expensive failures I can name sit in the gap between them.

The move

Two things stop a patient being given the wrong gas. One is a connector shaped so that the wrong cylinder cannot physically attach. The other is a label that capitalises the letters distinguishing two drug names that look alike. Same hospital, same accident, the same profession designing the fix. Only one of them works whether or not anybody is paying attention.

That pair carries the whole practical distinction. Removing an option is structural: it changes what can happen. Improving the odds is informational: it tells a careful person something true, then asks them to go on being careful at three in the morning in their eleventh hour. Both are design. Only one holds on a bad day — and the second degrades in a particular way, because a warning issued often enough and irrelevantly enough teaches people to click past it, which means a safety feature has trained its users to ignore it.

Then the inversion, which took me a while to accept. The stronger of the two has no trial behind it and never will, because you cannot randomise a plug that does not fit. The weaker one does: tall-man lettering performs well in controlled reading tests, with thin and mixed evidence on whether it prevents real dispensing errors. Rank design work by the strength of the evidence behind it and you will rank it close to backwards.

And the third piece: the artefact is not the intervention. A surgical checklist cut deaths from 1.5 per cent to 0.8 per cent across eight hospitals. The same checklist, made compulsory across roughly a hundred hospitals and 200,000 patients, did nothing measurable at all. Whatever carried the first result, it was not the paper. In edtech we ship the paper constantly, because the paper is the part that can be specified and bought.

Practised here, those three ideas become fairly blunt questions. What does this make impossible, rather than discouraged? What is the informational half asking a tired teacher to do reliably, and how often will it cry wolf? And what would have to be true in the school for the artefact to carry any of its effect — because if the answer is fifteen minutes of preparation that nobody has, the design is finished and the intervention has not started.

What it produced

The first thing it produced was an obligation to throw out my own sales material. The numbers designers reach for are not good. The best-known returns-on-design figures do not survive being looked up, and closest to home, a systematic review of usability and user experience in learning management systems found no study at all linking a usability improvement to a learning outcome. Not disproven. Never tested. I would rather say that early and loudly than have somebody else find it while I am mid-argument.

What replaced it was a better case that had been sitting in the education research the whole time. The argument for design in edtech does not rest on conversion rates or a multiple on spend. It rests on the variance being enormous, design accounting for a real share of it, and the downside not being symmetrical with the upside.

That asymmetry is the honest business case. Good design in this field produces modest, hard-won gains. Bad design produces total losses — devices bought at scale whose purchased curriculum was used by two schools out of sixty-nine, systems going live unable to tell a school which of its children have additional needs. Neither of those failed for lack of money. They had more money than almost any well-designed thing I can name. The case for design is closer to insurance than to growth, which is a harder thing to sell in advance and a much easier thing to justify afterwards.

The last thing it produced was a change in what I ask for. Not a seat at the table, a phrase that has never moved anything. A specific question, owned by somebody by name: who decided what this makes impossible? When nobody can answer, that is the finding.