A SHARING WITH EDTECH DESIGNERS

Zero is an average

By the end, you will be able to judge which edtech ideas are likely to help, which may quietly harm, and what to measure before they ship.

A note on shorthand. Educational technology becomes "edtech" from here, artificial intelligence becomes "AI", and effect sizes are given in standard deviations (SD) throughout.

I went back through this literature expecting it to support the case designers usually make. It mostly doesn't. What it supports instead is a better argument that almost nobody uses, and that is what follows.

Four claims run underneath. A pooled average can hide consequential variation. Contrasting programmes can open a question without identifying its cause. Pedagogy, implementation and context help explain why outcomes differ, and design must account for all three. And a fair amount of what our profession says in its own defence is badly evidenced, which is worth fixing before we ask anyone for a budget.

I'm not claiming good design reliably makes people learn more. Nobody has shown that, and I'm not going to pretend otherwise three sections after complaining about other people's overclaims. The claim is narrower and much harder to argue with.

Part one of three

What design in edtech is about

Forty years of research, the spread hiding behind the headline number, and why our own case for design is weaker than it looks.

• ONE What design in edtech is about
• TWO What design does, and what it costs
• THREE What UX designers in edtech do

Three parts follow. The first is about what the evidence in this field actually says, which is less flattering to everyone in it than the usual summary suggests.

The second is about consequences: what a badly designed programme costs, what a well designed one moves, and who decides both.

The third turns the whole argument around to face the person doing the work. The colour changes there, and that is deliberate.

§01

THE WAGER

TWO WHOLE PROGRAMMES · NOT A CONTROLLED PAIR

Which one worked?

Peru put laptops with children. Bridge International in Kenya put scripted lessons on teachers' tablets and monitored delivery.

A · PERU

A laptop for every child

• 319 rural schools
• A laptop placed with each child
• Teachers trained; measured at 15 months

B · KENYA

Tablets for the teachers

• Lessons scripted on the teacher tablet
• Teachers deliver; heads monitor
• Lottery allocation; measured over two years

FIG 01 — THE AVERAGE DESCRIBES NEITHER PROGRAMME

Peru. Nearly every child in 319 rural schools received a laptop. The machines arrived, teachers were trained and children used them. Reading and maths were measured after 15 months.

Kenya. Bridge International used tablets as a teaching system: the screen carried scripted lessons, teachers delivered them and school heads monitored adherence. Places were allocated by lottery from roughly 30,000 applicants, and outcomes were measured over two years.

These are two different programmes, not controlled versions of the same intervention. The comparison does not prove what caused the difference. It makes the question concrete: was value created by distributing hardware, or by designing what happened in the lesson?

Cristia et al. 2017 · Gray-Lobe et al. 2022

THE ANSWER

One of them did nothing at all

After 15 months, Peru showed no reading or maths gain. Over two years, the Bridge schools in Kenya produced an unusually large gain.

0.003

Peru · effect size in maths and reading

0.81

Kenya · unusually large effect size

Peru's effect was 0.003 standard deviations—statistically nothing. The confidence interval was tight enough to rule out anything above 0.11. Delivery was not the problem: the laptops arrived, teachers were trained and children used them.

The Bridge programme's effect was 0.81 standard deviations, above the 90th percentile of measured effects in international education. The gains appeared across schools and teacher profiles and were largest for the lowest-achieving children.

The contrast cannot isolate a cause because the programme models were different. It establishes the puzzle that the next section investigates.

§02

ZERO IS AN AVERAGE

Part one of three

§02

The average is hiding the argument

40 years of research, one headline number, and a spread underneath it that is rarely discussed.

Pool 40 years of research on technology in education and you get an effect of roughly 0.35 standard deviations (SD). That pooled summary is quoted constantly, but its usefulness for predicting an individual implementation is limited by the wide variation underneath it.

This section does three things. It shows the spread. It shows how the same review team found little difference when comparing media and a larger difference when comparing designed interaction. And it retires the question the sector has been asking since the 1980s.

Richard Clark settled the media-versus-method argument in 1983. The field has spent 40 years not absorbing it, which is why procurement exercises still compare devices.

40 YEARS OF RESEARCH

The average is a poor predictor of any one implementation

Pool the evidence and the mean is 0.35. The result that matters is the wide variation around it.

FIG 02 — THE POOLED MEAN HIDES LARGE VARIATION

Tamim and colleagues pooled 25 meta-analyses covering 1,055 primary studies and found a mean effect of 0.35 standard deviations (SD), with high heterogeneity. The heterogeneity is the finding, not a nuisance to be apologised for.

The pooled mean describes the literature, but it predicts an individual implementation poorly when the distribution behind it is this heterogeneous. The important question is what helps determine where an implementation lands.

That's the whole argument, and it's why the sector's favourite question has been unanswerable for 40 years. You can't get a useful number out of a distribution this wide by taking its mean.

Tamim et al. 2011

THE CLEANEST EXPERIMENT IN THE FIELD

Same method: compare media, then interaction design

The same review team pooled 232 media comparisons in 2004, then 74 comparisons of designed interaction in 2009.

FIG 03 — INTERACTION DESIGN SHOWED THE LARGER DIFFERENCE

In 2004 Bernard's team pooled 232 studies comparing distance learning with classroom teaching. The result was essentially zero, with the authors noting that plenty of online courses did much better and plenty did much worse.

Five years later the same team, using the same methods, stopped comparing online with classroom and started comparing online courses with each other, sorted by how student-to-student, student-to-teacher and student-to-content interaction had been designed. 74 studies. Weighted average effect on achievement: 0.38.

The media comparison was near zero; variation in instructional interaction was associated with a much larger difference. That is evidence that designed learning interactions matter more than delivery medium alone, not proof that every form of design causes learning.

It is also the cleanest contrast pair in the literature, and worth stealing for reasons the last section comes back to.

Bernard et al. 2004 · Bernard et al. 2009

EDUCATION ENDOWMENT FOUNDATION

Pedagogy is the learning method. Context is the system around it. Interface design can support or obstruct both, but it is not interchangeable with either.

It is not the technology per se that makes the difference; rather it is the pedagogy and the context.

EEF evidence review, 2019

Richard Clark said a version of this in 1983 and spent a decade being argued with. Media are vehicles. Whether anything useful is inside the vehicle is a separate question, and it's the one nobody was asking.

Kozma's reply in 1994 reframed rather than refuted it: ask how a medium's particular capabilities can be exploited, not whether the medium alone causes learning. Clark's closing line was "Media and method." He was right, and the field moved on without absorbing it.

The debate was settled and then ignored, which is why we're still running procurement exercises that compare devices.

Lewin et al. 2019 · Clark 1983 · Kozma 1994

THE REFRAME

Which design works, for whom, under what conditions?

Pedagogy, implementation and context create the variance. Design must account for it.

This is the pivot the whole argument turns on. Stop asking whether a category of technology produces learning on average. Start asking what separates the top of that distribution from the bottom, because the gap is enormous and it isn't random.

Design is not a single causal variable. It is the discipline of making pedagogy, implementation and context work together for a named learner. That narrower claim is the one the evidence can carry.

§03

BURN THE EVIDENCE

Part one of three

§03

Let me throw out the sales material

The case we usually make for design is badly evidenced.

Every argument that follows depends on the numbers holding up. So the numbers our own profession leans on have to go first, before anyone else takes them apart for us.

Saying so ourselves is worth more than any case study. It costs nothing and it buys everything that follows.

THE RULE EVERY CONSULTANCY QUOTES

The quoted IBM study does not exist

The familiar claim says faults cost 100× more after launch. Source trail: Bossavit and Wayne.

100×

the unsupported after-release slogan

0

traceable studies behind the quoted IBM rule

It's attributed everywhere to the IBM Systems Sciences Institute. Someone went looking. The trail ends at a 1987 textbook citing internal IBM course notes. The Institute was a staff training programme, not a research body, and no supporting data was ever published.

Hillel Wayne's summary: "There's one tiny problem with the IBM Systems Sciences Institute study: it doesn't exist."

Nielsen Norman Group, the most serious usability authority in the business, still repeats the number in its own return-on-investment report. That's the cleanest illustration available of how a citation chain launders an untraceable figure into authority.

A version of the claim does appear in a real IEEE paper by Boehm and Basili, so the honest line is that the popular slogan is a rumour, not that the underlying idea is false.

Bossavit, via Anderson 2021 · Wayne, n.d.

AND THE TWO BIG INDICES

Correlation is evidence, not causal proof

MCKINSEY

32-point revenue-growth correlation

• No causal test
• Dataset and company list unpublished
• Consultancy-authored

DESIGN MANAGEMENT INSTITUTE

211% index outperformance

• 16 companies chosen after success
• Apple dominates the result
• Advocacy-body authored

Both are published by parties with a commercial stake. McKinsey sells design consulting. The Design Management Institute (DMI) exists to advance design's economic standing. Neither has been independently replicated.

Mauro and Thurman's critique is the one to read: McKinsey cannot claim design actions measurably improve business performance using correlation analysis alone. Rumsey's is the blunter version, and it's the Apple point. Remove one company and the index largely collapses, while Netflix returned 6,330% over the same window without being a design exemplar.

Neither is disqualifying on its own. What's disqualifying is quoting them as though they settled something.

Sheppard et al. 2018 · Rae 2016 · Mauro & Thurman 2019 · Rumsey 2024

AND THE ONE CLOSEST TO HOME

The review found no test

Usability, satisfaction and task success are not learning outcomes.

190

papers screened

62

analysed in depth

0

linking a usability gain to a learning outcome

Nakamura and colleagues put it in plain words: "There were no studies relating the influence of the usability/UX improvement in the learning process." UX there is user experience, and the finding is that nobody has connected an improvement in it to an improvement in learning.

Worse, usability's own components barely agree with each other. Across 73 studies, task effectiveness and user satisfaction correlate at 0.164. If satisfaction doesn't track whether someone finished the task, it certainly doesn't track whether they learned anything.

This does not contradict the earlier finding about instructional interaction. It narrows the claim: evidence about learning design cannot be silently converted into evidence that any usability or interface improvement raises learning outcomes.

So the claim closest to home is untested rather than false. That distinction matters, and it is the one to make out loud before someone else makes it for you.

Nakamura et al. 2017 · Hornbæk & Law 2007

THE TURN

So what evidence survives?

I have used the hundred-times rule in a presentation. More than once.

Nobody ever asked me where it came from

Which is precisely how a number survives 40 years without a source.

None of this concedes the argument. It clears the ground. The strong case was sitting in the education research the whole time, while design people borrowed numbers from marketing.

And it matters more than it looks. Bring one of those figures into a room with a competent analyst and you've handed the sceptics their case permanently. In an organisation that already half suspects design of being decoration, you get to do that once.

Part two of three

What design does, and what it costs

What bad design costs, what good design moves, and the organisation that decides both before a designer arrives.

• ONE What design in edtech is about
• TWO What design does, and what it costs
• THREE What UX designers in edtech do

The first part argued that design is the variable. This one is about the size of the stakes attached to that claim, and it is the part that pays for the rest.

Two public post-mortems, three separate failure modes, and then the organisation that sets the constraints a designer works inside.

§04

WHAT BAD DESIGN COSTS

Part two of three

§04

What bad design costs

Two public post-mortems. Both make the case for early risk detection, not guaranteed growth.

The business case for design is usually pitched as growth. These cases support something duller and far more defensible: early risk detection.

The figure in this section is a conceptual risk model, not a measured distribution. Its point is that catching one expensive failure early can justify design work even when average gains are difficult to attribute.

Both cases here come from the same district, and neither failed for want of funding. They had more money than almost any well-designed thing I can name. What they lacked was a process that would catch a problem while it was still cheap to fix.

The second is the more instructive of the two. It shows an organisation actively destroying the only signal that could have saved it, and nobody involved intended that.

LOS ANGELES UNIFIED · IPAD PROCUREMENT

The solution was selected before fit was established

The district wrote a specification that favoured Apple, then committed $1.3 billion before it knew whether schools would use the curriculum.

$1.3bn

total programme

2 of 69

schools used the purchased curriculum

The decision. Los Angeles Unified planned to put iPads into schools. A federal review later found that its specification heavily favoured Apple and effectively excluded other options.

The purchase. The district committed about $1.3 billion and bought 43,000 devices at $768 each. The preloaded maths and English curriculum had been selected from samples rather than evidence of how it would work across the district.

The outcome. Only two of 69 schools used the purchased curriculum. Around 5% of students could access the software consistently. The district eventually recovered $6.4 million from Pearson.

The problem was not simply an expensive device. Procurement fixed the product before the district had established fit, access or a usable instructional model.

Newcombe 2015 · LA School Report · Instructure 2024

LOS ANGELES · A SECOND PROJECT · STUDENT RECORDS

Leadership called it “Feed the pig”: staff paid 25¢ whenever they said the old system had worked better.

Feed the pig.

A quarter went into “the pig” each time the comparison was made

The project. The same district built a new student-records system for an organisation serving hundreds of thousands of students. Engineers stress-tested it for only 1,200 concurrent users.

The warning. Staff were saying that the system it replaced had worked better. Leadership responded with a scheme called “Feed the pig”: every time someone made that comparison, they were fined a quarter.

The consequence. Criticism became something staff were trained not to voice. After launch, 197 of 201 schools reported that the system could not reliably identify students with disabilities.

The lesson is not the slogan. It is the sequence: weak testing produced warnings, leadership penalised the warnings, and serious failures survived into use.

Ask what it costs someone in your organisation to say the new thing is worse than the old one. If the cost is anything above zero, negative evidence will arrive late or not at all.

LAUSD Inspector General, via LA School Report

THE SHAPE OF THE RETURN

The honest business case is insurance

A risk-management framing: design can expose expensive failure while it is still cheap to change.

FIG 04 — CONCEPTUAL MODEL · DESIGN AS EARLY RISK DETECTION

The figure is a conceptual risk model, not an empirical distribution. The claim is not that good design always produces only modest gains or that bad design always produces total loss. It is that preventing a rare, expensive failure can justify design work even when average gains are hard to prove.

Neither Los Angeles project failed for lack of money. They had more money than almost any well-designed thing I can name. What they lacked was a process that would catch a problem while it was still cheap to fix.

Which reframes design investment as risk management rather than value creation. Less exciting, far more defensible, and the version that survives scrutiny.

§05

THREE DIAGNOSES

Part two of three

§05

Three diagnoses

Three errors, each with its own evidence.

The full argument has seven moves. Nobody retains seven. These three are the categories they fall into, sorted by which error they correct rather than by how important they sound.

None of them is about taste. None of them needs a bigger budget. Each one I've made more than once.

DIAGNOSIS ONE

Our instruments answer the wrong question

Every metric on the dashboard measures performance while the product is still there.

Completion, time on task, questions answered, streaks, satisfaction. All of it is performance-with-the-tool.

Capability once the tool is gone is a different variable, and the two can point in opposite directions, in the same students, in the same week. Almost nobody measures the second one.

ONE MODEL · TWO PROMPTS · 1,000 STUDENTS · SHOW OF HANDS

Predict before I show you

Switched on, A raised performance 48%. B raised it 127%. Then the AI was taken away.

A

Plain GPT-4

• No modification
• Students ask, it answers

B

Scaffolded GPT-4

• Worked solutions supplied
• Common misconceptions listed
• Told to hint, never answer

Commit to an answer before reading on. A, B, or no difference.

Roughly a thousand Turkish high school students, four maths lessons, three conditions including a control with no AI at all. Then a closed-book exam once the AI was removed.

The last section calls this "predict before you look". It works because being cheaply wrong in private is what builds calibration, and there is no cost to being wrong here.

Bastani et al. 2025, Proceedings of the National Academy of Sciences (PNAS)

THE REVEAL

The scaffolded version did not win. It broke even.

Control baseline: 0. Careful design removed the measured harm; it did not create a measured gain.

−0.054

Plain GPT-4 · worse than students who never had AI

−0.004

Scaffolded GPT-4 · no measured difference from control

FIG 05 — BOTH LOOKED LIKE WINS UNTIL THE AI WAS GONE

Be precise about the −0.004, because most write-ups aren't. The scaffolded tutor didn't beat the control. It neutralised the harm. Careful design bought back a deficit that careless design created.

Nothing differed between the two conditions except the prompt. Same model, same students, same four lessons. One version quietly damaged learning while making everyone involved feel more productive.

So build one measurement that happens when your product isn't in the room. Delayed, unaided, ideally on paper. It'll be the least flattering number you own and the only honest one.

Generative AI has taken the cost of producing educational material to roughly zero. When production is free, design is the only thing left that separates one product from another, and it also makes bad design cheaper to ship than it has ever been.

THE SECOND WAY OUR INSTRUMENTS LIE

Ease and learning come apart

Physics students were randomised between active and passive versions of the same material, both taught by trained instructors.

FIG 06 — THE FEELING RAN OPPOSITE TO THE FACT

What happened. Students in the active classes learned 0.46 standard deviations more. The same students reported learning 0.56 standard deviations less, rated the passive lecturers as better teachers and asked for more passive teaching.

Why it matters. The feeling moved as far in the wrong direction as the learning moved in the right one. Satisfaction described how fluently the lesson felt, not how much the students could later do.

The same warning appeared in an AI study. Interfaces intended to stop people following a wrong AI recommendation reduced overreliance from 64% to 48%. Users nevertheless rated those interfaces as more complex and trusted them less. The study supports the warning that preference can point away from protection; it does not tell us that every disliked interface is better.

A product team that optimises only satisfaction and engagement can therefore select the version that feels easier while missing the outcome that matters.

Deslauriers et al. 2019, PNAS · Buçinca, Malaya & Gajos 2021

STANDARDISED EFFECTS · 25 STUDIES

Cost with nothing to show for it

25 studies. Do hard-to-read fonts improve learning?

+0.52

longer study time

−0.43

lower confidence

−0.01

no recall gain

3,135 participants. Learners worked longer, felt worse, and learned exactly the same amount.

Robert Bjork, whose work on desirable difficulties gets cited to justify this sort of thing, is explicit about the limit. If a learner can't meet the difficulty successfully, it isn't desirable. It's just difficult.

Difficulty pays only when it's the difficulty of retrieving, choosing, or explaining. Everything else is tax.

Xie, Zhou & Liu 2018 · Bjork & Bjork 2020

THE DISTINCTION THAT RESOLVES IT

Remove interface friction. Preserve cognitive effort.

Remove effort that spends attention without producing thought. Preserve the effort of retrieving, choosing and explaining.

FIG 07 — ONE FRICTION IS WASTE, THE OTHER IS THE WORK

Interface friction is waste. Cognitive effort is the whole product.

Stating it is easy. Applying it is the hardest judgement in this work, because the two frictions feel identical from the inside. A student stuck on your navigation and a student stuck on your question both look stuck, sound frustrated, and generate the same support ticket.

The test that works is removal. Take the friction out and watch what's lost. If nothing is lost, it was waste. If the task suddenly feels effortless, you've probably just removed the learning.

Both failure modes are expensive and only one is visible. Strip everything and you ship a product that feels wonderful, tests beautifully, teaches nothing, and reports success on every metric you own. Add difficulty for its own sake and at least people complain.

I get this wrong regularly, and always in the same direction. Smoothing things is what I was trained to do, it feels like competence, and the metrics reward it immediately.

DIAGNOSIS TWO

We design the wrong object

The object is the learning activity—not the interface or device that delivers it.

Peru specified hardware; Kenya specified an instructional system. That is one important difference between the programmes, not the entire explanation for the distance between 0.003 and 0.81. The distance-learning reviews point in the same direction: comparing media produced little difference, while comparisons of designed interaction produced more.

This failure is almost invisible from inside a product team, because every artefact a product team produces is a screen.

TEST ONE · THE DELETE TEST

What remains when the product is removed?

Keep the learning activity. If no value remains, there was no learning design underneath the interface.

Remove the product and keep the activity. If something valuable remains, you designed a learning experience. If nothing remains, you built a delivery mechanism with nothing to deliver.

TEST TWO · THE SUBTRACTION TEST

Can you cut 40% without losing the argument?

Remove seductive detail, redundant narration and detached labels before adding anything else.

The best-evidenced principles in the entire learning-materials literature are subtractive. They tell you what to take out.

The seductive details effect is well replicated, and it names precisely the material a product team fights hardest to keep, because it's the material that demos well. Every release plan I've ever seen is additive.

So the standing question for a design review is this: what's the smallest version of this that still carries the idea?

Mayer & Fiorella 2014 · Noetel et al. 2022 · Sundararajan & Adesope 2020

DIAGNOSIS THREE

The average user does not exist

Three different failures follow: treatment varies, many people are never exposed, and gains land unevenly.

1 · Treatment heterogeneity. In education the same design can produce opposite results for people with different prior knowledge.

2 · Exposure. A design cannot help the people who never encounter or use it, however strong its measured treatment effect may be.

3 · Distribution. An average gain can hide where the benefit lands. In education the most consequential gains often come from lifting the bottom.

Taken together, these findings mean the learner must be named and the conditions of use made explicit. A single persona or average can conceal people who need opposite things.

PRIOR KNOWLEDGE · 176 EFFECT SIZES, 60 EXPERIMENTS

The same scaffolding, opposite effects

Across 60 experiments, the effect of high instructional support changed with what learners already knew.

FIG 08 — HELP A NOVICE, HANDICAP AN EXPERT

The pattern. Across 176 effect sizes, more instructional guidance helped learners with low prior knowledge by about half a standard deviation and harmed learners with high prior knowledge by nearly as much.

The mechanism. Support that reduces the burden on a novice can become unnecessary processing for someone who already has a working mental model. The evidence is a pattern across experiments, not one universal screen that always produces a sign flip.

The design decision. Either adapt support to prior knowledge, or name the learner and level you are designing for. “Our users” is too broad when different learners need opposite amounts of guidance.

Tetzlaff et al. 2025

AND THE PERSON WHO NEVER ARRIVED

Assume many people will not open it

Two randomised trials offered an AI reading tutor to primary-school children. About half never logged in.

~50%

never logged on once

2–5 min

weekly use among those who did

30 min

the recommended dose

The intended treatment. Primary-school children were offered an AI reading tutor for a recommended 30 minutes a week.

What actually happened. Roughly half never logged in. Those who did averaged only two to five minutes a week. The lead researcher concluded: “We never really got close enough to the dosage needed to find out.”

What the trials establish. They did not deliver enough use to answer whether the tutor worked at the intended dose. They do not, from the evidence given here, establish why use was low.

The design implication. The delivery routine is part of the intervention. A product that depends on regular use needs a credible answer to who schedules it, who supervises it and where it fits—not only another engagement feature.

Related, and routinely ignored. Teacher time isn't free. Dutch secondary teachers get roughly 15 minutes of paid preparation per lesson. A design that assumes an hour of setup has already failed, whatever it did in the pilot.

Robinson et al. 2026 · Janssen et al. 2017

AND WHERE TO POINT THE GAIN

Aim at the floor, not the ceiling

Tutor CoPilot gave human tutors live suggestions: +4 mastery points overall, +9 among the lowest-rated tutors.

FIG 09 — THE GAIN LANDS AT THE FLOOR, NOT THE CEILING

The intervention. Tutor CoPilot gave human tutors real-time suggestions that they could accept, edit or ignore. It cost about $20 per tutor per year.

The result. Students gained four percentage points of topic mastery overall. Students taught by the lowest-rated tutors gained nine points.

The mechanism. Transcripts showed that tutors using the tool asked students to explain their reasoning about 10% more often and gave generic encouragement about 10% less.

Kenya's result had the same shape. Uniform across schools and teacher profiles, largest gains for the lowest-achieving children.

Consumer product design optimises the delight case. That instinct travels badly. The wins here come from lifting the bottom of the distribution, which demos poorly and matters enormously. Nobody has ever been promoted for it either.

Wang et al. 2025

THE FULL SEVEN

Seven moves, three underlying errors

WRONG INSTRUMENTS

• 01 Measure after removal
• 02 Protect useful effort

WRONG OBJECT

• 03 Design the lesson
• 04 Cut before adding

FALSE AVERAGE PERSON

• 05 Name the learner
• 06 Design for exposure
• 07 Lift the floor

FIG 10 — SEVEN SYMPTOMS, THREE UNDERLYING ERRORS

01Measure it after you take it away Performance-with-the-tool and capability-without-it are different variables, and they can point in opposite directions in the same students in the same week. Completion, time on task, questions answered, streaks: all of it is the first one. Build one measurement that happens when your product is not in the room. Delayed, unaided, ideally on paper. It will be the least flattering number you own and the only honest one.

02Know which friction you are protecting A student stuck on your navigation and a student stuck on your question both look stuck, sound frustrated, and generate the same support ticket. The test that works is removal. Take the friction out and watch what is lost. If nothing is lost it was waste. If the task suddenly feels effortless you have probably just removed the learning.

03Design the lesson, not the screen Peru specified the hardware. Kenya specified the lesson. Delete your product and keep the activity design underneath it: if anything of value is left, you designed something. If the activity was never designed, you built a delivery mechanism with nothing to deliver. The failure is invisible from inside a product team, because every artefact a product team produces is a screen.

04Cut before you add The best-evidenced principles in the entire learning-materials literature are subtractive. Coherence: take out the interesting extras. Redundancy: stop narrating the screen. Contiguity: put the label on the thing rather than below it. The seductive details effect names precisely the material a team fights hardest to keep, because it is the material that demos well. Every release plan I have seen is additive.

05Name your learner, or adapt Heavy scaffolding helps learners with low prior knowledge by half a standard deviation and harms learners with high prior knowledge by nearly as much. The sign flips. So you get two honest options: adapt, which is expensive and usually done badly, or name the learner out loud and accept that you are wrong for everyone else. Most products quietly pick the novice, then act surprised when their strongest students stop showing up.

06Design for exposure Roughly half the children in two randomised trials of an AI reading tutor never logged on once, and those who did averaged two to five minutes a week against a recommended 30. Take-up is often the binding constraint, not quality, and it is rarely solved by a feature alone. It gets solved by a person, a routine, or a slot on a timetable.

07Aim at the floor Four percentage points of topic mastery overall, nine for the lowest-rated tutors, at about $20 per tutor per year. Kenya had the same shape: uniform across schools, largest gains for the lowest achievers. Consumer product design optimises the delight case and that instinct travels badly. Education is a variance problem, and the wins come from lifting the bottom of the distribution.

One and two correct the wrong instruments. Three and four correct the wrong object. Five, six and seven correct the false average person.

None of it is about taste. None of it needs a bigger budget. It needs you to want the unflattering number more than the good demo.

Which is where the whole thing usually dies. Most designers already understand this list. Ignorance is rarely the obstacle.

§06

THE BUILDING

Part two of three

§06

You can do all seven perfectly and still lose

What stops us is the building.

This section matters most if you are coming into public education from outside, from a vendor, an agency, or private-sector product work. The craft transfers. The conditions do not.

The organisation exists to reduce variance in decision-making, and design's contribution is to introduce doubt exactly where doubt is least welcome. I don't have a way around that, and I'm suspicious of anyone who says they do.

NATIONAL TEACHER REGISTRATION SERVICE

Its predecessor was built in six months while the policy it had to implement was still changing.

Built in six months as highly complex policy was still being designed.

UK Department for Education, on a national teacher registration service

FIG 11 — DESIGN ARRIVES AFTER THE DECISIONS

The constraint. A British government team inherited a national teacher-registration service whose predecessor had been built in six months while the underlying policy was still being designed.

The outcome. The service launched with inaccurate records, unsustainable support demand and manual workarounds for every user group.

The bill. Building the replacement took two years. Half the team was occupied with technical debt while also trying to create the new service.

The point is not that no design work occurred. It occurred after policy, time and delivery constraints had already fixed much of the result. The team was repairing consequences it had not been allowed to prevent.

UK e-University · a separate demand failure. The programme spent £50 million and recruited 900 students against a target of 5,600. A parliamentary committee found that it had conducted no formal research into either the level or nature of demand and had continued without questioning its approach. In this case, asking prospective users was not a step in the decision process.

DfE Digital 2026 · HC 205, 2005

THE MOVE WITH THE MOST REACH

Change the exam and lessons change

Assessment, curriculum, procurement and timetable are design objects with system-wide reach.

The assessment format, the curriculum specification, the procurement rules, the timetable. Those are design objects, currently being designed by people who don't know they're designing.

The GOV.UK Design System is the visible version of the same idea. More than 1,200 services use it, and when the accessibility rules changed, its team published the guidance centrally rather than leaving 1,200 teams to solve it separately. Deciding once, for everyone, is what a ministry is for.

Write a procurement specification carelessly and you've selected for the wrong product for a decade. Los Angeles wrote one that a federal review found favoured a single vendor, then spent $1.3 billion acting on it.

TWO FAILURES, OPPOSITE CAUSES

Permission has more than one form

inBloom lost public consent for a working data platform. The Bridge network from the opening Kenya case lost legal permission to operate some schools.

INBLOOM · SOCIAL CONSENT

Technical ambition without consent

• $100m data platform
• Parent opposition over data use
• Closed within two years

BRIDGE · LEGAL LICENCE

Outcomes did not override the law

• 10 of 12 Busia schools ordered closed
• Unregistered teachers
• Facilities and environmental compliance

inBloom · social consent. inBloom was a $100 million platform for school data. The technology worked, but parents did not want their children's records held in it. The company described the objection as “generalized public concerns about data misuse” and closed within two years. It had designed the data infrastructure without securing the relationship and consent required to sustain it.

Bridge · legal permission. Bridge International was the school network behind the Kenya instructional programme introduced at the beginning. Its measured learning results were exceptionally strong. Separately, Kenya's High Court upheld the closure of 10 of 12 Bridge schools in Busia County over unregistered teachers, unsuitable facilities and the absence of an environmental assessment.

The cases fail for different reasons. inBloom lost the consent of the families whose data it held. Bridge's learning result did not override the legal conditions for operating schools.

A company that ships a bad release loses users. A ministry that ships a bad release loses consent, and consent doesn't iterate back. Public trust is the one asset a better next version cannot recover.

Which makes "who could kill this, and have we spoken to them" a design question rather than a communications one.

Bulger et al. 2017 · Gray-Lobe et al. 2022 · Right to Education Initiative 2017

WHAT THE JOB ASKS OF THE PERSON

Three capabilities the institution demands

UNDERSTAND THE INSTITUTION

• 01 Bureaucratic literacy
• 02 Value unglamorous work

WORK WITH OTHER EXPERTISE

• 03 Be the second-best expert
• 04 Respect changed work

MAINTAIN EVIDENCE AND TRUST

• 05 Test our own claims
• 06 Carry bad news
• 07 Wait for invisible payoff

UNDERSTAND THE INSTITUTION

01Bureaucratic literacy Read the institution as fluently as you read an interface: who drafts, who clears, what people are measured on, and when the money is settled.

02Value unglamorous work The highest-value work may be a form template, data definition or tender clause. Take the boring surface seriously even when it never enters a portfolio.

WORK WITH OTHER EXPERTISE

03Be the second-best expert Your value sits in the seam between curriculum, policy, engineering and delivery. Be useful to people who know more than you about their subject.

04Respect the work you change Remember the limited preparation time available to teachers. Treat resistance as information about working conditions, not automatically as an adoption problem.

MAINTAIN EVIDENCE AND TRUST

05Test our own claims first Hold accurate numbers, including those that make design look worse. Removing a weak statistic buys more credibility than another case study.

06Carry bad news steadily Say when the new thing is worse than the old one, from a position of low formal authority, without becoming impossible to hear.

07Wait for invisible payoff Some of the best work is a disaster that did not happen: hard to prove, lightly rewarded and absent from most reports.

The craft is the easy part. This is the list I would hire against, and I do not clear all of it.

Part three of three

What UX designers in edtech do

The person doing the work: their own learning, the job itself, and what the week looks like.

• ONE What design in edtech is about
• TWO What design does, and what it costs
• THREE What UX designers in edtech do

Everything before this described a field. This part describes a job, and it starts with the person holding it rather than with the product.

Your own learning first, because the evidence about designing for learners applies to the people doing the designing. Then the role, then the week.

§07

APPLY IT TO YOURSELF

Part three of three

§07

Apply it to yourself

Everything above, aimed at the person doing the designing.

The colour changes here, and that is deliberate. Everything before this was the field. This is us.

If the evidence about designing for learners is worth anything, it applies to the people doing the designing. We are learners too, and most of us develop ourselves using precisely the methods the research says do not work.

Three things follow. What you can drill, which is craft. What you can only train sideways, which is judgment. And what keeps you in the room long enough for either to matter.

None of it needs a course, a certificate, or permission from an employer. Almost nobody does it.

ONE LAST THING

Reading this can feel like learning. That is not the same as being able to use it.

Look away from the page and reconstruct the three diagnoses from memory before continuing.

The colour just changed, and that is deliberate. Everything up to here was the field. This is us.

Most of us develop ourselves using precisely the methods the research says don't work. Conference talks. Design newsletters. A book about a methodology. Case studies on Medium. All of it feels productive, and nearly all of it is recognition.

POOLED FINDINGS · DIFFERENT STUDY SETS

Recognition is not recall

Not a head-to-head comparison. First, reconstruct the three diagnoses without looking.

0.36

recognition · "yes, that makes sense"

0.81

free recall · closing the tab and rebuilding it from memory

The feeling of understanding tracks how fluently the material went down. It doesn't track whether you can use any of it on Tuesday. That is the same finding as the physics result, aimed at the person doing the designing.

So: read a case, close the tab, write the mechanism from memory. What was decided, what it cost, what the alternative was. Do it again a week later.

Don't over-engineer the interval. Expanding schedules test no better than flat ones, so any spacing you'll actually stick to beats the clever one you won't.

Rowland 2014 · Latimier et al. 2021

HOW JUDGMENT IS ACTUALLY TRAINED

Judgment grows through close contrasts

Mixing examples helps when learners must distinguish similar categories. The useful work is comparison, not variety.

CONCRETE EXAMPLE

Plain versus scaffolded GPT-4

• Same model and students
• Prompt and support changed
• Unaided outcome changed

THE METHOD

Keep close contrasts

• Name the common surface
• Name the decisive difference
• Record the evidence

The finding. Interleaving means mixing categories during practice instead of studying one category at a time. It helped when the categories were easy to confuse: learners comparing paintings by different artists improved by 0.67 standard deviations because they had to notice the differences between styles. It did not help with foreign vocabulary, where the items did not require the same kind of visual discrimination.

The mechanism. The useful work is deciding which feature separates two similar cases. Simply adding variety is not the lesson.

The practice. Keep near pairs and write down what looked the same, what actually differed and which outcome the evidence supports. Plain versus scaffolded GPT-4 is the clean example here: the students and model stayed the same; the prompt changed; the unaided result changed.

A library of admired examples can tell you what good work looks like. A library of close pairs can help you explain why one case worked and another did not.

Brunmair & Richter 2019

AND THIS WAS ONE LONG PAIR

Four you already have

Four cases were deliberately paired. Only the closer pairs isolate a specific difference.

• 01 Peru / Kenya · programme model
• 02 Plain / scaffolded GPT-4 · prompt design
• 03 inBloom / Bridge · form of legitimacy
• 04 Bernard 2004 / 2009 · unit of comparison

The pairs are an organising device, not four equivalent experiments.

Peru and Kenya share a broad aim but use different programme models: hardware placed with children versus scripted lessons delivered by teachers. The pair opens a design question; it does not isolate one cause.

Plain and scaffolded GPT-4 hold the model, students and lessons steady while changing the support in the prompt. This is the closest experimental pair in the deck.

inBloom and Bridge show two different permissions a public service needs. inBloom lost parents' consent to hold student data; Bridge schools lost legal permission over staffing and facilities. This is a conceptual comparison, not an outcome comparison.

Bernard 2004 and 2009 use similar review methods but ask different questions. Comparing delivery media produced a near-zero mean. Comparing the design of learning interactions produced a 0.38 mean effect. The contrast suggests that what learners do matters more than the channel alone.

Build the same kind of file from your own work. Pair a project that landed with a similar one that did not, then record what stayed constant, what changed and what evidence supports the explanation.

Keep the pairs. That file is worth more than any framework you'll ever adopt.

NO COURSE, NO PERMISSION, NO BUDGET

What to do from Monday

• 01 Reconstruct instead of rereading
• 02 Keep contrast pairs
• 03 Measure capability after the product is removed

EVERY STUDY

01Reconstruct instead of rereading Read a case, close the tab, and write the mechanism from memory: what was decided, what it cost, and what the alternative was.

02Ask three questions How many people? Compared with what? Measured how, and when?

03Predict before you look Write down the outcome you expect and why. Calibration needs a record of being wrong.

EVERY WEEK

04Cut something by 40% Take a screen, deck or policy document and remove 40% without losing the argument.

05Keep contrast pairs Record what was identical, what differed, and which outcome the evidence supports.

06Sit at the bottom of the distribution Spend time with the teacher who has 15 minutes, the student who never logged on, and the school with the oldest hardware.

EVERY SIX MONTHS

07Measure after removal Return to old work and test what people can do without the product in the room.

08Retire scaffolding you have outgrown Review the process that once supported you and ask whether it now carries the work or merely carries you.

None of this needs a course, a certificate, or permission from an employer. And do not confuse suffering with growth: working late is the disfluent font of professional development.

AND THE HONEST CAVEAT

I manage maybe half of it consistently.

Roughly the hit rate to expect from anyone handing out advice about their own field

A list of virtues with no admission attached is a job advertisement. One admission makes it an argument.

I reconstruct rather than reread, most weeks. I keep contrast pairs, and the file is genuinely useful. I am poor at writing predictions down before I look, and worse at going back to see what happened to work I shipped three roles ago, which is the one I would most benefit from.

The point of saying so is not modesty. It is that advice delivered without a hit rate is unfalsifiable, and this whole argument has been for wanting the unflattering number.

§08

THE ROLE

Part three of three

§08

What the job actually is

Six duties that follow once design is the variable rather than the finish.

Everything above describes a field. This describes a job. If design is the variable that separates the two programmes at the start of this, then the person responsible for design is accountable for an outcome, not for a screen.

Six duties follow. Three of them restate moves already made here, aimed now at the person doing the work. Three are new, and the newest is the one about evidence: our own profession's growth case is weaker than the loss case, and the loss case is the one worth making.

The order matters. Measurement comes first because it decides what everything else is optimising. Research comes second because you cannot adjudicate a friction you have not watched happen. The rest follow from those two.

WHAT FOLLOWS FROM THE EVIDENCE

You own the variable, so you own the outcome

In most industries design is one input among several. Here it is the input the evidence keeps landing on.

That is a heavier claim than the profession usually makes for itself, and it cuts both ways. If design explains the difference between a programme that moves nothing and one that moves 0.81, then a flat result is not the technology's fault.

Accountability for an outcome is also the strongest position a designer can argue from. It is far more defensible at a budget meeting than a usability score nobody has linked to learning.

The uncomfortable corollary is that the programmes that worked were designed too, often by people who were never called designers. Curriculum leads, teacher trainers and the person who wrote the lesson script. The job is to be worth having in that room, not to claim it.

SIX DUTIES

The job, in six lines

• 01 Own the measure, and refuse the one that flatters you
• 02 Research the lesson, not the screen
• 03 Adjudicate friction: strip waste, protect the effort that is the work
• 04 Be the risk function rather than the growth function
• 05 Aim at the floor, where the gain actually lands
• 06 Design the constraints, because you arrive after the decision

01Own the measure Satisfaction, engagement, completion and SUS all track the feeling of learning. Insist on a delayed, unaided measure taken with the product removed.

02Research the lesson The object of study is a lesson inside a timetable, not a person using software. Observe it. Self-report is not valid for the outcome that matters.

03Adjudicate friction Interface friction costs attention and produces nothing. Cognitive effort is what produces the learning. Most usability work strips both because both look like problems in a test.

04Be the risk function Good design produces modest, arguable gains. Bad design produces total loss. The honest business case is early detection, not growth.

05Aim at the floor Design for the least confident teacher, the oldest device and the worst connection, and you capture most of the available effect.

06Design the constraints Procurement criteria, the definition of done, the accessibility standard and the assessment format all sit upstream of the product and outrank it.

Three of these restate moves already made here, turned around to face the person doing the work. Three are new: the research duty, the risk framing, and the refusal of the wrong measure.

None of them is about the screen, which is the point. A designer whose job description stops at the interface has accepted a version of the role the evidence does not support.

THE BOUNDARY

What the role is, and what it is not

IS

Accountable for an outcome

• Owns the difference between feeling and learning
• Decides which friction survives
• Reads the procurement document
• Recruits at the floor

IS NOT

Responsible for the surface

• Making the platform prettier
• Owning the design system, though someone should
• Representing users in the abstract
• Neutral facilitation of whatever the room wants

The right-hand column is not a list of unimportant work. It is a list of work that does not need this argument to justify it, and that quietly becomes the whole job when nobody can say what a good outcome would look like.

The test is simple. If the role could be described without reference to whether anyone learned anything, it has drifted.

THE FAILURE MODE

A designer who optimises for how it feels is making it worse

The feeling of learning runs against learning. Optimising for the feeling and calling it user-centred is the one mistake this role is uniquely placed to make.

This is the sharpest edge of the whole argument, and it is genuinely uncomfortable. Every instinct trained in consumer product design points the wrong way here.

It does not license making things unpleasant. Difficulty is not the same as effort. It means the pleasantness of the experience is not evidence about the outcome, and should never be reported as if it were.

Deslauriers et al. 2019 · Buçinca et al. 2021

THE HONEST CASE

Do not reach for the hundred-times rule

The growth case for design is badly evidenced. The loss case is public, documented and much larger.

FIG 12 — BAD DESIGN LOSES MORE THAN GOOD DESIGN GAINS

The next time somebody asks what design is worth in this sector, the strongest available answer is the two programmes this session opened with. Point at the gap and ask which of the two they think they are currently building.

That argument survives scrutiny. A hundred-times multiplier traced to a textbook citing internal course notes does not.

Anderson 2021 · Mauro & Thurman 2019

§09

THE PRACTICE

Part three of three

§09

How the work actually runs

The research, the artefacts and the weekly rhythm the duties imply.

Principles are cheap. This section is the part that costs something: what gets done on a Tuesday, which methods survive contact with a school, and which of the standard toolkit has to be replaced rather than adapted.

Three traditions are in play and they do different jobs. Design thinking frames an ill-defined problem well. Service design supplies the multi-actor view. Design-based research supplies the part that decides whether any of it taught anyone anything.

Access is the constraint nobody plans for. Safeguarding clearance takes weeks, consent runs through the school, and the academic calendar decides everything. Nothing useful happens in the first fortnight of term, in an assessment window, or in the last week before a holiday.

Brown 1992 · Design-Based Research Collective 2003 · McKenney & Reeves 2018

THE UNIT OF ANALYSIS

Research the lesson, not the screen

A lesson, delivered by a teacher, inside a timetable, over a network, to 30 people with different prior knowledge. The software is one touchpoint in that service.

Most UX research here studies a person using software, which is a component of the thing that determines the outcome rather than the thing itself.

That single change of object reorders the whole toolkit. It moves the work from product research toward service research, and it makes the classroom rather than the lab the place the evidence comes from.

THREE TRADITIONS, THREE JOBS

Use each for what it is good at

• DT Design thinking frames an ill-defined problem, and settles nothing about efficacy
• SD Service design supplies the multi-actor, multi-touchpoint view a product mindset misses
• DBR Design-based research decides whether the thing taught anyone anything

Empathise, define, ideate, prototype and test will tell you whether people want the thing and can operate it. It will not tell you whether it worked. Bolt the efficacy question onto the end of every cycle or the loop closes on the wrong criterion.

Service design contributes the actor map and the blueprint. Edtech separates the person who pays, the person who delivers and the person who benefits, and that conflict is the design problem rather than a complication of it.

A persona flattens exactly the thing you most need to see, because it turns a structural conflict between a ministry, a school leader, a teacher and a learner into a personality with a photograph.

Design Council 2019 · Stickdorn & Schneider 2011 · Shostack 1984

THE SPINE OF THE PROCESS

A cycle that ends in a principle

Analyse with practitioners. Develop from theory. Test in the real setting. Reflect, and state what transfers.

FIG 13 — THE CYCLE ENDS IN A PRINCIPLE, NOT A RELEASE

Design-based research grew out of the learning sciences in the early 1990s for this exact problem: how to design an intervention and find out whether it taught anything, in a classroom rather than a lab.

Put it beside the Double Diamond and the shape is familiar. Two things are added. The solution is grounded in theory that already exists, and the test environment is the classroom, so the evidence produced is about learning rather than about task completion.

Brown 1992 · Collins 1992 · McKenney & Reeves 2018

BEFORE ANY WORKSHOP

Start in the literature

Retrieval practice, spacing, interleaving, coherence and worked examples all have meta-analytic support. Rediscovering them in a sprint wastes a term you cannot get back.

Discovery normally starts because nobody knows what users need. Education is unusual: a great deal is already known, at a standard of evidence UX rarely gets near.

Week one is desk research against the learning sciences, and it produces a memo. What the evidence says works for this kind of learning, at what effect size, under what conditions, and where it is thin. That memo is the evidence base for the brief.

Generative research then goes after what the literature cannot tell you, which is this school, this timetable, this teacher and this constraint. The two are complements, and running the second without the first is how teams end up paying to rediscover the testing effect.

Rowland 2014 · Latimier et al. 2021 · Brunmair & Richter 2019 · Noetel et al. 2022

THE ARTEFACT THAT EARNS ITS KEEP

Blueprint one lesson, end to end

From the teacher opening a planning tab the evening before to the marks landing in the information system.

FIG 14 — MOST FAILURES SIT BELOW THE LINE OF VISIBILITY

Service blueprinting separates what the user sees from what makes it possible. Front stage is the learner's view. Back stage is planning, setting, chasing and marking. Support processes are the timetable, login provisioning, device charging, the network and the information system.

The failures in this session all sit below that line. A curriculum nobody could reach and a system that could not report which children had disabilities are both support-process failures, and neither is visible in an interface review.

Shostack 1984 · Bitner et al. 2008

IN THE ROOM, NOT IN A LAB

Five things to record in a lesson

• 01 How long from the bell to the first minute of instruction
• 02 Every point at which the lesson stops, and what stopped it
• 03 What the teacher does in the first ten seconds after it stops
• 04 How many learners are off-task, and at which minute
• 05 Every workaround. The spreadsheet beside the platform is the finding

Interviews tell you what a teacher believes about a lesson. Contextual inquiry tells you what happened in it, on the master-and-apprentice footing that method describes.

Schools already run learning walks and lesson observations. Use their protocol rather than importing a research instrument that makes everyone perform.

Recruit at the floor while you are at it. Standard practice recruits typical users, and a study run with your three most enthusiastic pilot teachers predicts almost nothing about the fourth school.

Beyer & Holtzblatt 1998

THE CADENCE

What the week looks like

• WEEK One lesson observed with the laptop shut. One teacher conversation about what broke. One evidence question answered
• CYCLE Three to five sessions in context, fixing the prototype between them. One blueprint update. One thing removed
• TERM One delayed measure of what shipped last term. One read of the next specification while it can still change
• NEVER A satisfaction score reported as an outcome. A discovery sprint that ignores 40 years of evidence

Fixing the prototype between sessions rather than after them matters more here than anywhere else. In a school you may get three sessions in a term, and spending all three collecting the same failure is not rigour.

Cutting is the only reliably safe design move in a domain where extraneous material has a measured cost. If a cycle produces no removals, it probably produced no decisions either.

Access is the real constraint. Safeguarding clearance takes weeks, consent runs through the school and often the parents, and the data protection assessment has to exist before the first session. Build a standing relationship with three or four schools rather than recruiting fresh each time, because the setup cost dominates everything else.

Medlock et al. 2002 · Sundararajan & Adesope 2020

§10

CLOSE

BACK WHERE WE STARTED

Good intentions did not distinguish them

0.003

Peru

0.81

Kenya

Both programmes were well funded and intended to improve learning. The two effect sizes neither reveal intent nor isolate the cause of the difference.

The return to them is a reminder that good intentions are not a measurement strategy. The job is to name the design, learner and conditions, then test what remains when the product is gone.

Skip measurement, and you are experimenting without knowing the consequences.

With other people's learning.

With public money.

And calling it innovation.

Four essays sit behind this: Zero is an average, The right side of the average, Designing where you don't decide, and Apply it to yourself. The full seven moves, the reading list and every source quoted here are in those.

THANK YOU

What did I get wrong?

Challenge the argument, ask for a source, or continue the conversation.

• WEB geraldajam.com
• EMAIL me@geraldajam.com

SINGAPORE · 1.35°N 103.82°E

If you disagree with any of it, that is the right response to an argument that spends five minutes claiming the evidence in this field is weaker than it looks. I'd rather hear where I have it wrong than have it agreed with politely.

Source trail

References

The works used to build the presentation and its reading-mode article. Entries follow APA 7th edition; links point to the publication, report or source page where available.

1. Anderson, T. (2021, July 22). Everyone cites that “bugs are 100x more expensive to fix in production” research, but the study might not even exist. The Register. https://www.theregister.com/2021/07/22/bugs_expense_bs/
2. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), Article e2422633122. https://doi.org/10.1073/pnas.2422633122
3. Belward, B. (2026, June 5). Move fast, then fix things—What we learned building Register early career teachers. DfE Digital, Data and Technology. https://dfedigital.blog.gov.uk/2026/06/05/move-fast-then-fix-things-what-we-learned-building-register-early-career-teachers/
4. Bernard, R. M., Abrami, P. C., Borokhovski, E., Wade, C. A., Tamim, R. M., Surkes, M. A., & Bethel, E. C. (2009). A meta-analysis of three types of interaction treatments in distance education. Review of Educational Research, 79(3), 1243–1289. https://doi.org/10.3102/0034654309333844
5. Bernard, R. M., Abrami, P. C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P. A., Fiset, M., & Huang, B. (2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74(3), 379–439. https://doi.org/10.3102/00346543074003379
6. Beyer, H., & Holtzblatt, K. (1998). Contextual design: Defining customer-centered systems. Morgan Kaufmann.
7. Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008). Service blueprinting: A practical technique for service innovation. California Management Review, 50(3), 66–94. https://doi.org/10.2307/41166446
8. Bjork, R. A., & Bjork, E. L. (2020). Desirable difficulties in theory and practice. Journal of Applied Research in Memory and Cognition, 9(4), 475–479. https://doi.org/10.1016/j.jarmac.2020.09.003
9. Boehm, B. W., & Basili, V. R. (2001). Software defect reduction top 10 list. Computer, 34(1), 135–137. https://doi.org/10.1109/2.962984
10. Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2(2), 141–178. https://doi.org/10.1207/s15327809jls0202_2
11. Brunmair, M., & Richter, T. (2019). Similarity matters: A meta-analysis of interleaved learning and its moderators. Psychological Bulletin, 145(12), 1029–1052. https://doi.org/10.1037/bul0000209
12. Bulger, M., McCormick, P., & Pitcan, M. (2017). The legacy of inBloom. Data & Society. https://datasociety.net/library/the-legacy-of-inbloom/
13. Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), Article 188. https://doi.org/10.1145/3449287
14. Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459. https://doi.org/10.3102/00346543053004445
15. Clough, C. (2015, April 15). Just in: LAUSD seeking refund for iPads with Pearson software. LA School Report. https://www.laschoolreport.com/just-in-lausd-seeking-refund-for-ipads-with-pearson-software/
16. Collins, A. (1992). Toward a design science of education. In E. Scanlon & T. O’Shea (Eds.), New directions in educational technology (pp. 15–22). Springer.
17. Cristia, J., Ibarrarán, P., Cueto, S., Santiago, A., & Severín, E. (2017). Technology and child development: Evidence from the One Laptop per Child program. American Economic Journal: Applied Economics, 9(3), 295–320. https://doi.org/10.1257/app.20150385
18. Design Council. (2019). Framework for innovation. https://www.designcouncil.org.uk/our-resources/framework-for-innovation/
19. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. https://doi.org/10.3102/0013189X032001005
20. Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251–19257. https://doi.org/10.1073/pnas.1821936116
21. Government Digital Service. (n.d.). Accessibility strategy. GOV.UK Design System. https://design-system.service.gov.uk/accessibility/accessibility-strategy/
22. Gray-Lobe, G., Keats, A., Kremer, M., Mbiti, I., & Ozier, O. (2022). Can education be standardized? Evidence from Kenya [Working paper]. SSRN. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4129184
23. Hornbæk, K., & Law, E. L.-C. (2007). Meta-analysis of correlations among usability measures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 617–626). Association for Computing Machinery. https://doi.org/10.1145/1240624.1240722
24. House of Commons Education and Skills Committee. (2005). UK e-University (HC 205). The Stationery Office. https://publications.parliament.uk/pa/cm200405/cmselect/cmeduski/205/205.pdf
25. Instructure. (2024, June 24). New LearnPlatform by Instructure report finds increases in more unique digital tools accessed by K–12 institutions, students, and teachers. https://www.instructure.com/press-release/new-learnplatform-instructure-report-finds-increases-more-unique-digital-tools
26. Janssen, F. J. J. M., Könings, K. D., & van Merriënboer, J. J. G. (2017). Participatory educational design: How to improve mutual learning and the quality and usability of the design? European Journal of Education, 52(3), 268–279. https://doi.org/10.1111/ejed.12229
27. Kozma, R. B. (1994). Will media influence learning? Reframing the debate. Educational Technology Research and Development, 42(2), 7–19. https://doi.org/10.1007/BF02299087
28. Latimier, A., Peyre, H., & Ramus, F. (2021). A meta-analytic review of the benefit of spacing out retrieval practice episodes on retention. Educational Psychology Review, 33, 959–987. https://doi.org/10.1007/s10648-020-09572-8
29. Lewin, C., Smith, A., Morris, S., & Craig, E. (2019). Using digital technology to improve learning: Evidence review. Education Endowment Foundation. https://files.eric.ed.gov/fulltext/ED612157.pdf
30. Mauro, C. L., & Thurman, P. W. (2019, January 25). Critical analysis: Why McKinsey’s “The business value of design” report ultimately misstates the value of UX design and design overall. Medium. https://medium.com/@PulseUX/why-mckinseys-the-business-value-of-design-report-ultimately-misstates-the-value-of-ux-design-dd6fc0b236ab
31. Mayer, R. E., & Fiorella, L. (2014). Principles for reducing extraneous processing in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 279–315). Cambridge University Press.
32. McKenney, S., & Reeves, T. C. (2018). Conducting educational design research (2nd ed.). Routledge.
33. Medlock, M. C., Wixon, D., Terrano, M., Romero, R., & Fulton, B. (2002). Using the RITE method to improve products: A definition and a case study. Usability Professionals Association Conference.
34. Nakamura, W. T., Oliveira, E. H. T., & Conte, T. (2017). Usability and user experience evaluation of learning management systems: A systematic mapping study. In Proceedings of the 19th International Conference on Enterprise Information Systems (Vol. 3, pp. 97–108). SCITEPRESS. https://doi.org/10.5220/0006363100970108
35. Newcombe, T. (2015, May 11). A cautionary tale for any government IT project: L.A.’s failed iPad program. Governing. https://www.governing.com/archive/gov-tablets-los-angeles-ipad-apple-schools.html
36. Nielsen, J., Berger, J. M., Gilutz, S., & Whitenton, K. (n.d.). Return on investment (ROI) for usability (4th ed.). Nielsen Norman Group. https://opus.bsz-bw.de/fhdo/files/2166/ROI_for_Usability_4th_Edition.pdf
37. Noetel, M., Griffith, S., Delaney, O., Harris, N. R., Sanders, T., Parker, P., del Pozo Cruz, B., & Lonsdale, C. (2022). Multimedia design for learning: An overview of reviews with meta-meta-analysis. Review of Educational Research, 92(3), 413–454. https://doi.org/10.3102/00346543211052329
38. Rae, J. (2016). Design Value Index exemplars outperform the S&P 500 Index (again) and a new crop of design leaders emerge. Design Management Review, 27(4), 4–11. https://doi.org/10.1111/drev.12040
39. Right to Education Initiative. (2017, February 17). Kenyan court upholds the closure of Bridge International Academies schools over failure to respect standards. https://www.right-to-education.org/fr/node/893
40. Robinson, C. D., Gormley, D., Ribeiro, A. T., & Loeb, S. (2026). Access is not enough: Human support improves engagement with AI tutoring [Working paper]. EdWorkingPapers.com. https://doi.org/10.26300/pz7p-p388
41. Romo, V. (2014, November 19). Long-awaited LAUSD report calls MiSiS “grossly inadequate.” LA School Report. https://www.laschoolreport.com/long-awaited-lausd-report-calls-misis-grossly-inadequate/
42. Rowland, C. A. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin, 140(6), 1432–1463. https://doi.org/10.1037/a0037559
43. Rumsey, R. (2023, December 16). You may have seen this image before [LinkedIn post]. LinkedIn. https://www.linkedin.com/posts/ryanrumsey_you-may-have-seen-this-image-before-maybe-activity-7141805603970039808-55bm
44. Sanders, E. B.-N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. CoDesign, 4(1), 5–18. https://doi.org/10.1080/15710880701875068
45. Sheppard, B., Sarrazin, H., Kouyoumjian, G., & Dore, F. (2018, October 25). The business value of design. McKinsey & Company. https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-business-value-of-design
46. Shostack, G. L. (1984). Designing services that deliver. Harvard Business Review, 62(1), 133–139.
47. Stickdorn, M., & Schneider, J. (2011). This is service design thinking: Basics, tools, cases. BIS Publishers.
48. Sundararajan, N., & Adesope, O. O. (2020). Keep it coherent: A meta-analysis of the seductive details effect. Educational Psychology Review, 32, 707–734. https://doi.org/10.1007/s10648-020-09522-4
49. Tamim, R. M., Bernard, R. M., Borokhovski, E., Abrami, P. C., & Schmid, R. F. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81(1), 4–28. https://doi.org/10.3102/0034654310393361
50. Tetzlaff, L., Simonsmeier, B., Peters, T., & Brod, G. (2025). A cornerstone of adaptivity: A meta-analysis of the expertise reversal effect. Learning and Instruction, 98, Article 102142. https://doi.org/10.1016/j.learninstruc.2025.102142
51. Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2025). Tutor CoPilot: A human-AI approach for scaling real-time expertise (EdWorkingPaper 24-1054). Annenberg Institute at Brown University. https://edworkingpapers.com/sites/default/files/ai24_1054_v2.pdf
52. Wayne, H. (2021, July 19). I ****ing hate science. Computer Things. https://buttondown.com/hillelwayne/archive/i-ing-hate-science/
53. Xie, H., Zhou, Z., & Liu, Q. (2018). Null effects of perceptual disfluency on learning outcomes in a text-based educational context: A meta-analysis. Educational Psychology Review, 30(3), 745–771. https://doi.org/10.1007/s10648-018-9442-x

End of the talk

Back to geraldajam.com