Holding the frame lightly

Every analysis is a frame, and every frame leaves something out. This move runs a conclusion through more than one model, asks how each measure will change what it measures, and names who sits outside the frame and pays if it is wrong. It ends with a commitment held as partial, and the trigger that would reopen it.

Complexity: the four moves

Move 4 of 4 · Keeps the analysis honest

Check what your model leaves out, and who pays if it is wrong

Time: 2 hours
People: 4–8, including someone who disagrees with the plan and, where possible, someone affected by it
You need: The conclusion or figure to be tested, the three-models grid and the measure sheet, printed
You leave with: The conclusions that survive several models, a note on how each measure will be gamed, who is outside the frame, and a working frame with its revisit trigger

Needs a confident facilitator

The idea

What the move is, and why it works

What the move is

This move is a check on the analysis itself. It comes after a plan, a figure or a recommendation has started to look settled. It asks three things of it. What would other credible models say? How will people respond to our measures? Who is outside our frame, and who pays if we are wrong?

It ends with a commitment. Holding a frame lightly does not mean holding none. You choose one working frame, say what it leaves out, and write down what would make you look again.

The idea behind it

Every model simplifies, so every model is wrong somewhere. Using several, built on different simplifications, shows which conclusions survive across them. Those are the ones to act on. The rest are contested, and the plan should say so.

The analyst is also inside the situation. A target, a rating or a dashboard is a move, and the people it describes will answer it. Any measure used for decisions starts to change what it measures. And a frame decides who counts. The people outside it usually carry the cost of its mistakes, and are the last to be asked.

Why it works

It replaces “is this analysis right?” with a question a school can answer: is it sound enough to act on, and safe enough to be wrong about? That moves attention from precision to consequences, which is where a school has some control.

It also catches the error confident analyses make most often. They are right about the part they measured and silent about the part they did not. A second model, or one person from outside the room, usually finds that part in minutes.

When

When to reach for it, and when not to

Use it when

An analysis has come out confident and neat.
A target, rating or dashboard is about to be introduced or reported upward.
A decision affects people who were not in the room.
A figure looks good and nobody believes it.

Skip it when

The decision is small and easy to undo. A premortem and one question about who pays will do.
The group has not yet committed to anything. Holding a frame lightly needs a frame to hold.
Nobody in the room can afford to disagree with the person who owns the plan. Fix that first, or bring in an outside chair.

Before

Before anyone is in the room

Choose the one conclusion, figure or plan to test, and write it as a sentence.
Find two other people or groups who would read the situation differently, and get their reading, in their words, before the meeting.
List every measure the plan uses or reports, and who sees each one.
Invite one person the plan affects who has not been consulted, or bring their words if they cannot come.
Print the three-models grid and the measure sheet.

Steps

How to run it

7 steps, about 2 hours in all. The questions in quotation marks are meant to be read out as they are.

01
Name the working frame and its measure
10 min
Write the conclusion you are testing as one sentence, and the measure that supports it. Read both aloud. Ask the group what picture of the situation the sentence assumes.
Ask the room
- What would have to be true for this sentence to be right?
- What does our main measure count, exactly?
Watch for Defending the frame before it has been named. This step is description only.
02
Run it through two more models
30 min
Set out at least two other credible readings of the same situation, from a different discipline or a different position: a teacher, a parent, an allied educator, a student. For each, say what it thinks is going on, what it would count as good, and what it leaves out. Then mark what survives across all three.
Ask the room
- How would someone who teaches this every day explain what is happening?
- How would a parent at home read the same figure?
- Which of our conclusions would all three readings accept?
- Which conclusion survives only in our own model?
Watch for Relativism. Several models is not permission to pick the one that suits. Some readings are better supported than others. The point is to find what survives.
03
Measure the measure
20 min
For each measure, ask how people will behave once it counts, and what it will stop showing once they do. Assume every measure people can see will change what they do. Decide for each one: keep it, change it, stop reporting it, or keep it with a note saying what it no longer shows.
Ask the room
- How would a sensible person move this number without changing anything real?
- Who is already doing that?
- What will this number stop showing once people have learnt it?
Watch for Cynicism. “Every measure gets gamed, so measure nothing” hands judgement to whoever is loudest.
04
Hold a premortem
15 min
Ask everyone to imagine it is a year from now and the plan has failed. Each person writes the most likely reason, alone and in silence. Then read them out, one each, around the room.
Ask the room
- It is a year from now and this has failed. What happened?
- Which of these reasons did we already know about?
Watch for Polite reasons. If nobody names a person or a relationship, the premortem was too safe.
05
Ask who is outside the frame
25 min
Use boundary questions. Who is this plan for? Who decided its shape? Whose knowledge counts in it? Who is affected and has no say? Then ask who carries the cost if the analysis is wrong, and whether they will find out before you do.
Ask the room
- Who does this plan serve, in practice?
- Who is affected by it and was not asked?
- If we are wrong, who carries the cost, and how soon would we know?
- Whose knowledge would have changed this if we had asked for it?
Watch for A list of the people left out, with nothing done about any of them. Pick one and change something.
06
Test the action against being wrong
10 min
Ask whether the action would still be sound if the analysis were wrong. Could you undo it? If not, could you afford it? Would you find out quickly? If the answer is no to all three, the plan needs a smaller first step, not more analysis.
Ask the room
- If we are wrong, can we undo this?
- If we cannot undo it, can we afford it?
- How long before we would know?
Watch for Precision used as comfort. A more detailed version of the same model does not answer this question.
07
Commit, provisionally, with a trigger
10 min
Choose one working frame. Say in a sentence what it leaves out. Write down the event or the evidence that would make you revisit it, and who will notice. Then act on it.
Ask the room
- Which frame will we act on for now?
- What does it leave out that we will keep watching?
- What would make us look again, and who will tell us?
Watch for A review that is minuted and changes nothing. The output has to touch a measure, a decision or a report.

Worksheets

Two sheets to print and fill in

Each one prints on its own sheet of A4. The grey line under each is an example from the worked situation.

Complexity · Holding the frame lightly · Worksheet

Three models, one situation

Shows which conclusions survive across several readings, and which rest on one.

Conclusion being tested

Date

Who was in the room

Whose model	What it says is going on	What it would count as good	What it leaves out	Which of our conclusions it accepts

Example The plan’s | Feedback in the agreed format means a feedback culture | 95 per cent of books compliant | Whether any student acted on the feedback | Only its own

geraldajam.com/work/toolkit

Complexity · Holding the frame lightly · Worksheet

Measure the measure, and who pays

Checks one measure for what it will teach people to do, and names who carries the cost if the frame is wrong.

Measure

Reported to

Date

Revisit if

What the measure counts, and what it is meant to show

How a sensible person would move it without changing anything real

What it will stop showing once people have learnt it

Who is affected and was not asked

If we are wrong: who carries the cost, and how soon we would know

Our working frame, what it leaves out, and the trigger that reopens it

Example 96 per cent of books compliant. Moved by writing longer comments for the book check. It has stopped showing whether students used the feedback, and it is still reported to the cluster superintendent.

geraldajam.com/work/toolkit

Worked example

The 96 per cent, and Aisyah’s Emerging

One school changes how it assesses

In Term 2, Week 5 the dashboard showed 96 per cent of books compliant. Nobody in the school trusted it. The same week the cluster newsletter cited the school as an exemplar of “feedback culture”. A week earlier, at the first checkpoint, Aisyah had frozen in the timed English paper and been rated Emerging, and Mdm Rohani had complained through the app. Mrs Lim ran the move with Mr Ravi, the allied educator for learning support, an HOD who disliked the plan, and Aisyah’s form teacher.

Three models of the same figure The plan’s model said feedback in the agreed format is a feedback culture, so 96 per cent is success. Mr Ravi’s said feedback is whatever changes the next lesson, so the figure counts the wrong thing; his whole-class feedback had halved his marking and would fail the book check. The parents’ model, from the class parents’ chat group in Term 1, Week 6, said that without weighted assessments nobody knows where children stand. One conclusion survived all three: the figure does not tell anyone what students know. The 95 per cent target survived only the plan’s own model.
The ratings as one model of Aisyah The group put Aisyah’s record on the table. The rating said Emerging. Her class work all semester said strong. Her record on the unrecorded quizzes showed steady learning. The checkpoint measured writing against the clock, which is where she freezes. Each was a model of Aisyah, and the rating was the only one her mother had seen. The allied educator added that students still building their English, including two who joined from overseas this year, were rated Emerging in nearly everything, because the ratings judged written explanation. It was the same fault at a larger scale.
Measuring the measures The compliance figure was moved by longer comments written for the book check, and had stopped showing whether students used feedback. The ratings were moving too. After Mdm Rohani’s complaint, a teacher had quietly rated borderline students up. So the scale was now being bent by who complains, which favours families with the time to do it. The group did not find a measure people could not learn to perform for. It agreed to check each one more often.
Who pays if the frame is wrong The boundary questions named Aisyah and Mdm Rohani first. Mdm Rohani works rotating shifts, had read the parents’ chat thread at 3am, and could neither pay for tuition nor get to a weekday Meet-the-Parents session. Students rated on writing they cannot yet do in English came next. Then teachers, who had lost a weekend to book-check comments. The cluster superintendent’s office sat outside the school’s frame and inside its measures: the cluster’s framework asked for the compliance figure, and its newsletter had now praised it.
A provisional commitment The working frame became “ratings and feedback that tell a student and a parent what to do next”. It leaves out comparison between schools, which the cluster still wants. The trigger to reopen it is the guidance on reporting for lower secondary, still unpublished. In Term 2, Week 8 the Saturday morning Meet-the-Parents slot let Mdm Rohani see Aisyah’s quiz record, and she saw learning the rating had not shown. In Term 2, Week 10 the scale parents see was cut to three points, and Mrs Lim proposed dropping the format check from book checks.

What came out of it

Two changes inside the school and one proposed outside it. The scale parents see went to three points, and the format check is proposed for removal. Ms Tay, who had watched the semester from the cluster superintendent’s office, proposed that the cluster stop collecting the twice-a-term ratings. The awkward part is the figure. The cluster superintendent’s office still wants the compliance number, and the newsletter now stands behind it. The school reports a figure it no longer steers by, and the scale staff see is still five points. Holding the frame lightly made the school honest about what its numbers show. It did not make the numbers go away.

Signs

How to tell it is working, and when it has gone hollow

Working

A conclusion is dropped because it survived only one model.
A measure is changed, stopped or reported with a note before the next reporting date.
Someone outside the room is asked, and their answer changes a decision.
The revisit trigger is written down with a name next to it.

Gone hollow

Plurality becomes a reason never to conclude.
A paragraph about the authors’ position is added to the report and nothing else changes.
The review is minuted and the target goes ahead unchanged.
The people who pay are listed and none of them is asked.

Variations

Other ways to run it

In 45 minutes

Run three steps only: the premortem, measure the measure on the one figure that will be reported, and one boundary question, “who pays if we are wrong?” End by writing the trigger.

On your own

Before you report a figure, write two other readings of it: one from a colleague who disagrees, and one from a parent. Then ask how you would move the figure without changing anything real. If you can think of a way, someone is probably already doing it.

With students

Show a class a measure that describes them, such as a rating or a target. Ask what students do to it, and what it misses about them. Their answers are a measure-the-measure review, done for you.

What to do with what you made

Change one measure before the next reporting date, or report it with a note on what it no longer shows. Give the revisit trigger to someone who will notice it. Then redraw the profile from the first move, because acting on the situation will have moved its dials.

Profiling the situation Back to the start: redraw the dials now that you have acted.The overlay Seven checks to run on any plan, dashboard or risk register.The analyst inside Why every measure is also a move, followed through the semester.

Tools

Tools, templates and guides worth having

Published elsewhere, and worth the time.

Method
Klein’s premortem (Harvard Business Review, 2007)
Imagine the plan has failed and write why. Ten minutes, and the cheapest test of being wrong there is.
Method
Ulrich’s boundary questions, from Critical Heuristics of Social Planning (1983)
Twelve questions on who a plan serves, who decides, whose knowledge counts and who is affected without a say. Step 5 uses four of them.
Book
Page, The Model Thinker (2018)
Many-model thinking as a habit, with the models set out for non-specialists.
Book
Koretz, The Testing Charade (2017)
How test-based accountability teaches schools to move the numbers. Measuring the measure, with school examples.
Book
Christodoulou, Making Good Progress? The Future of Assessment for Learning (2016)
Why one assessment struggles to serve formative and summative purposes at once. Directly relevant to ratings that try to do both.

Reading

Where it comes from

Start with the first on the list.

Page, S. E. (2018). The Model Thinker. Basic Books.
Many-model thinking as a practice.
Levins, R. (1966). The strategy of model building in population biology. American Scientist, 54(4), 421–431.
Robustness analysis, and “the intersection of independent lies”.
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90.
Why indicators distort what they measure once they count.
Ulrich, W. (1983). Critical Heuristics of Social Planning. Haupt.
The twelve boundary questions.
Jasanoff, S. (2003). Technologies of humility. Minerva, 41(3), 223–244.
What institutions owe when their models are partial.

Singapore