Keep your tools. Run seven checks over them.
The overlay does not replace a strategic plan, a dashboard or a risk register. It adds seven questions to them, one for each contribution, and changes what they produce. This page sets out the checks, what they add to nine common tools, five tests of robustness, and what happened when one school ran its assessment plan through all seven.
A layer, not a replacement
Complexity thinking adds most value not by replacing existing tools but by adding a layer to them. The overlay is that layer. It has seven checks, one for each contribution, and they can be applied to almost any analysis.
That matters in practice. A school already has an improvement plan, a risk register, a dashboard and a way of consulting parents. Asking staff to drop those for a new method costs time and goodwill, and the new method usually ends up as one more document. The overlay works on the tools a team already uses. The plan stays a plan. It just has to answer seven more questions before anyone signs it.
The practical test
No other form of thinking supplies all seven checks, and most supply none. That is the clearest practical evidence that complexity thinking contributes something distinct. An analysis that has passed through the overlay is materially different from one that has not, whatever framework it started from.
The checks
Seven checks, one per contribution
Each check is a question to put to an existing tool. None needs a new framework.
| Check | The question | From |
|---|---|---|
| Relations check | What relationships, between actors, between systems and between contexts, does this analysis not represent, and which of them could change the result? | 1. Relations before things |
| Disposition check | What is the situation already inclined to do without us, and are we working with or against that propensity? | 2. Dispositions before predictions |
| Plurality check | What would two other credible models or perspectives conclude, and which of our conclusions survive across all three? | 3. Plural models before the right model |
| Reflexivity check | How will the people and systems we are analysing respond to our framing, metrics and moves, and how will that change what we observe? | 4. The analyst inside |
| Conditions check | Are we specifying outcomes we cannot control? Which conditions, constraints and safe-to-fail probes would make good outcomes likelier and teach us fastest? | 5. Conditions before outcomes |
| Optionality check | Which options does this plan spend, which commitments are irreversible, and which signposts and triggers will tell us to change course? | 6. Optionality before optimisation |
| Responsibility check | Who and what sits outside our boundary, who bears the cost of our framing, and whose knowledge have we not consulted? | 7. Responsibility for framing |
On the tools you have
What each common tool assumes, and what the overlay adds
| Tool | What it assumes | What the overlay adds |
|---|---|---|
| Strategic plan | The future can be specified and worked back from | A direction-not-destination statement. Adaptive pathways with signposts and triggers. A probe portfolio for the most uncertain areas. |
| SWOT | Strengths, weaknesses, opportunities and threats are stable attributes | Relational and dispositional analysis: which factors interact, and what is the system already inclined to do? |
| Risk register | Risks are independent and quantifiable | The type of incertitude for each risk. Coupling between risks. Creeping and transboundary risks. The options being spent. |
| Theory of change / logic model | Inputs cause outputs cause outcomes, in sequence | Feedback loops. Emergent outcomes. Claims of contribution rather than attribution. Developmental evaluation. |
| KPI dashboard | Indicators measure performance without changing it | A reflexivity review of each indicator. Warm-data narratives alongside the metrics. Early-warning indicators for slow variables. |
| Design sprint | The user and the problem are bounded | Second-order and system-level effects. Boundary critique. Sensing after launch for emergent use. |
| Scenario planning | Futures are driven by external forces | Present dispositions. Actors’ reflexive responses to the scenarios themselves. Links to probes and options. |
| Business case / cost–benefit analysis | Value is the expected return on a chosen option | Option value. Reversibility. Robustness across scenarios. |
| Stakeholder map | Stakeholders are a list with interests and influence | Relationships among stakeholders. Levels of trust. The legitimacy dial. Who is missing from the map. |
Robustness
Robust enough to act on, safe enough to be wrong about
Robust analysis may be less precise and less confident than conventional analysis. It is also far less likely to fail catastrophically. It changes the decision-maker’s question from “is this analysis right?” to “is this analysis robust enough to act on, and safe enough to be wrong about?”
| Test | The question | Grounding |
|---|---|---|
| Across models | Do the conclusions survive different models of the system, built on different simplifications? | Levins, 1966; Page, 2018 |
| Across scenarios | Do the recommended actions perform acceptably across a wide range of plausible futures, not only the expected one? | Lempert et al., 2003 |
| Across perspectives | Do the conclusions hold when examined from the standpoint of differently placed stakeholders and knowledge traditions? | Bartlett et al., 2012; Funtowicz & Ravetz, 1993 |
| Across scales | Do the conclusions hold at the levels above and below the one analysed, local and system-wide, short run and long run? | Gunderson & Holling, 2002; Simon, 1962 |
| Across being wrong | If the analysis is wrong, is the resulting action reversible, affordable or recoverable, and will we find out quickly? | Arrow & Fisher, 1974; Klein, 2007; Sarasvathy, 2001 |
Worked example
The school’s assessment plan, through all seven checks
The plan as it stood on the first day of term, taken through each check in order. The findings come from what happened over the term.
Relations check
What it found
The plan treated each department and each student as a separate unit with a rating. The semester ran on ties it did not show. Students compared ratings in the corridor and had a list of “harsh” subjects by Term 1, Week 4. A class parents’ chat group turned Developing into a fall and sent some families to tuition centres. And heads of department read the first book check through the triple-marking policy that drove two teachers out of English. The verdict landed on a relationship that was already damaged.
What changed
Mrs Lim drew a second map, of who talks to whom about assessment: the parents’ chat, the corridor, the allied educator for learning support, Mr Ravi and the teachers already trying whole-class feedback. Book-check findings stopped being shared with the whole staff. She went to the humanities department meeting instead of sending the report. It helped, a little. English still agreed to only one of the three probes.
Disposition check
What it found
The school was already inclined to do several things without the plan. Students and teachers were both inclined to satisfy whatever counted. Green pen became “I will use more evidence”, and feedback was written for the book check, which is why compliance rose so fast and meant so little. Staff were inclined to hear any marking demand as the old triple-marking policy. Mr Ravi had started whole-class feedback in Term 1, Week 5, outside the policy, and halved his marking time. The plan pushed against the first tendency and ignored the rest.
What changed
The plan stopped trying to raise compliance and started working with what was already moving. Whole-class feedback became one of three probes in Term 1, Week 10, alongside a fortnightly quiz that is never recorded and students annotating their own work. The two staff learning sessions were cut to one, and the second Wednesday afternoon went to teachers showing each other feedback that saved time and still changed the next lesson.
Plurality check
What it found
The plan rested on one model: more written feedback in books means more learning, and a rating tells a parent where a child stands. Mr Ravi’s model was different. Feedback works when it changes the next lesson, so the lever is the teacher’s planning, not the student’s book. The allied educator for learning support and Mdm Rohani held a third. A rating reads what a student can show in timed writing, and that is shaped by language, nerves and home. Only a few conclusions survived all three. The compliance figure says little. What happens in the next lesson says more. Mdm Rohani saw it at the Meet-the-Parents session: Aisyah’s quiz record showed learning her rating did not. The 95 per cent target survived only the first model.
What changed
In Term 2, Week 10 Mrs Lim proposed dropping the format check from book checks. The revised plan names its three readings and says which actions rest on which, so that when one reading is shown to be wrong, everyone can see what else falls with it.
Reflexivity check
What it found
Almost every measure the school introduced changed the thing it measured. The book check produced feedback written for the book check, longer comments and a lost weekend. Green pen turned into ritual: “I will use more evidence”, some of it written by a chatbot. The scale taught students which subjects were harsh, and after the Term 2, Week 4 complaints one teacher quietly rated borderline students up. Even the school’s own fix came under pressure: one class asked for the quizzes to count. When the cluster newsletter cited the school for its “feedback culture”, the 96 per cent figure acquired a reputation to protect.
What changed
Each indicator now gets a short check before it is used: what would a sensible person do to move this number without changing anything real? Quiz results stay with the teacher, read for misconceptions and never recorded, though leaders keep asking for the data. Nobody pretends the new measures are immune. They are just checked more often.
Conditions check
What it found
Ninety-five per cent of books in the agreed format was an outcome the school could not control without corrupting it. The outcomes that did matter were students who know what to work on next and do it, and ratings that tell a parent something true. They depended partly on the promised guidance on reporting for lower secondary, which had still not arrived by the end of the semester.
What changed
The target became a direction: every student should know what to work on next and be able to show they have worked on it. Each probe got criteria for spreading or stopping it. Whole-class feedback met them and spread to two more departments, each of which changed it. The quiz continued, unrecorded. Annotation has not yet shown enough to spread or stop. The scale parents see was cut to three points. The quiz record also gives parents something to look at, whatever the promised guidance says.
Optionality check
What it found
The plan’s largest irreversible commitment was a three-year contract for the tracking system, signed for a discount before anyone knew how the scale would behave. Term 1, Weeks 3 and 4 showed the cost. The scale measured against the end of the year, departments read it two ways, and the system held every rating on five points. At the Term 2, Week 6 review the supplier offered a further discount to extend to five years and lock the five-point scale in.
What changed
Mrs Lim took a one-year extension at a higher price. The plan now sets out what would justify a longer deal: the promised guidance on reporting for lower secondary, departments using the scale the same way, and a year of parents reading the three-point version. The finance committee still thinks the saving was worth having, and Mrs Lim has had to defend the decision to the principal. Nothing about that argument is settled.
Responsibility check
What it found
The plan’s boundary ran round the school day and the marked book. Outside it sat families paying for tuition centres that set practice papers, shift-working parents who could not come to weekday meetings and read the chat group at 3am, and students like Aisyah, strong in class and frozen by a timed paper. Students still building their English were rated Emerging in nearly everything, because the ratings judged written explanation. They bore the cost of a framing they had no part in. The cluster superintendent’s office sat outside the boundary too, and its framework had helped set the target in the first place.
What changed
The Meet-the-Parents session gained a Saturday morning slot, and Mdm Rohani came for the first time in two years. The allied educator for learning support joined the group reworking the plan. Ratings for students still building their English are now checked against classwork and the quiz record before they go home. The gap between families who can pay for tuition and families who cannot is named in the plan and not solved. The cluster’s request was beyond the school’s reach, but Ms Tay, having watched the semester, proposed that the cluster stop collecting twice-a-term ratings.
Before and after
Same school, same goal, a different plan
The overlay did not make the plan right. It made it robust enough to act on, and safe enough to be wrong about, which was as much as the semester allowed.
The plan, as written
- Replace the term-by-term weighted assessments in Secondary 1 to 3 with two larger checkpoint assessments a year, one each semester.
- Every marked piece of work carries a written next step, and students answer it in green pen within a week.
- Each student rated twice a term in every subject on a five-point scale, from Emerging to Exceeding, and shown to parents through the school’s parent app.
- A three-year contract for the tracking system that holds the ratings, signed for a discount.
- Two sessions on feedback in the Wednesday afternoon staff learning slot, and a book check in every department each term to make sure the policy is followed.
- A target of 95 per cent of exercise books showing feedback in the agreed format by the end of the year, reported with the ratings to the cluster superintendent, as the cluster’s framework asks.
The plan, after the overlay
- A direction instead of a target: every student knows what to work on next and can show they have worked on it. No school-wide compliance target.
- Whole-class feedback in science, geography and two more departments, each adapting it. Reviewed each term against agreed signs for spreading or stopping it. Written next steps where they help the student, not on every piece to satisfy a check.
- A ten-minute quiz each fortnight, read by the teacher for misconceptions and never recorded. One class has already asked for it to count, and leaders keep asking for the data. Both requests are declined, for now.
- Two checkpoints a year, read alongside classwork and the quiz record. Ratings for students still building their English are checked against both before they go home.
- Book checks discuss feedback with departments, not grade them. The format check is proposed for removal. English has joined one probe, not three.
- Parents see a three-point scale and staff keep five. A one-year contract extension. A longer deal depends on three signposts: the promised guidance on reporting for lower secondary, departments using the scale the same way, and what parents make of the shorter one.
- Meet-the-Parents sessions on a Saturday morning as well as on weekdays. The gap between families who pay for tuition and practice papers and those who do not is acknowledged, not closed.
- The compliance figure still reported twice a term to the cluster superintendent’s office, because its framework still asks for it, with a note on what the figure counts and why the school no longer steers by it.
Reading
Where it comes from
Start with the first on the list.
Levins, R. (1966). The strategy of model building in population biology. American Scientist, 54(4), 421–431.
The origin of robustness analysis: trust the conclusions that independent models agree on.
Lempert, R. J., Popper, S. W., & Bankes, S. C. (2003). Shaping the next one hundred years: New methods for quantitative, long-term policy analysis. RAND.
The case for choosing actions that perform acceptably across many futures, not the best action for one.
Haasnoot, M., Kwakkel, J. H., Walker, W. E., & ter Maat, J. (2013). Dynamic adaptive policy pathways: A method for crafting robust decisions for a deeply uncertain world. Global Environmental Change, 23(2), 485–498.
Signposts and triggers worked out in detail, the method behind the optionality check.
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90.
Why an indicator used for decisions starts to distort what it measures, the problem the reflexivity check exists for.
Ulrich, W. (1983). Critical heuristics of social planning: A new approach to practical philosophy. Haupt.
Boundary critique, the main tool of the responsibility check.
Snowden, D. J., & Boone, M. E. (2007). A leader’s framework for decision making. Harvard Business Review, 85(11), 68–76.
Probe, sense, respond, the short version of the conditions check.
Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford Press.
How to evaluate a plan that is meant to change as it runs.
Klein, G. (2007). Performing a project premortem. Harvard Business Review, 85(9), 18–19.
A cheap way to ask, before acting, how the plan will turn out to have been wrong.
Unsettled
What I would still argue about
The questions this page does not settle.
- How would a school know the overlay improved its decisions, and not only made them slower and more careful? The checks are easier to state than to evaluate, and most of the evidence for tools like these is case illustration.
- Seven checks take time a busy team does not have for every decision. Which of them can be skipped on a small, reversible choice, and which never can, is not yet clear.
- The school’s probes came under the same pressure. A week after the quizzes started, one class asked for them to count, and leaders wanted the results on a spreadsheet. It is not known whether there is any form of sensing people cannot learn to perform for, or only forms that take them longer to learn.
- The overlay worked inside the school and could not change what the cluster asks for. How far up a school’s responsibility check should reach, and what it can do when it gets there, is still open.