The contribution that puts the person holding the clipboard back in the picture. In a human system, the analyst’s framing, measures and actions enter the situation and change it. So an analysis has to account for its own effects as well as everyone else’s.
How will our framing, metrics and moves change what we observe?
Every analysis of a human system is also an intervention in it. A target changes the behaviour it was set to track. A published forecast changes the outcome. A dashboard teaches people what counts. This contribution moves the analyst from the edge of the picture into it, and asks how our framing, metrics and moves will change what we go on to observe. Its roots are second-order cybernetics, Stacey’s work on managers and Campbell’s law on indicators. It treats gaming and performativity as ordinary features of the system, to be planned for. It is strongest where measures carry consequences and people can see them. It is weakest when it shrinks into a statement of position that changes nothing.
Outside school
A city starts posting hygiene ratings on restaurant doors, from zero to five. The idea is simple. Diners can see which kitchens are clean, and kitchens have a reason to get cleaner. Within a few years the average score rises. Some of that is cleaner kitchens. Some of it is kitchens learning what the inspector looks at: the fridge temperature log filled in every day, the cleaning rota on the wall, the folder of certificates by the door. The rating has become partly a measure of readiness for inspection.
Campbell described the pattern in the 1970s. The more a number is used to make decisions about people, the more it bends the activity it was meant to observe. Strathern’s version is the one that stuck: when a measure becomes a target, it stops being a good measure. Diners adapt too. Once most doors show a five, a three stops meaning “fine” and starts meaning “avoid”, so owners with a three put their effort into appeals and re-inspections. The sticker has changed what the number means as well as what kitchens do.
The inspector is not standing outside any of this. Her checklist, her visiting pattern and the council’s decision to publish are all moves the kitchens respond to. If she starts checking under the fridges, the floor under the fridges gets cleaned. The extractor hood she never looks at does not. Second-order cybernetics made the general point decades ago. The observer is part of the system observed.
None of this makes the sticker useless. It makes it an intervention, to be judged as one. The useful question moves from whether the rating is accurate to what it is teaching kitchens to do, and what it will be teaching them in three years. Schools know the same pattern from league tables. A school that quietly admits fewer pupils likely to pull its results down is responding sensibly to how it is ranked. Nobody designed that. The table did.
The term
Every page follows the same Singapore secondary school through the same semester, Term 1 and Term 2. The school is changing how it assesses, both of learning and for it: fewer weighted assessments, more feedback, and ratings that are meant to say something. The five people and the events of the semester stay fixed. Only the contribution reading them changes.
All five, interconnected
The five threads interleaved across the twelve weeks, so the hand-offs between them are visible. Pick a person to follow their term on its own. The people are illustrative, and the same five appear on every contribution’s page.
Term 1, Week 3
Mrs Lim, at the dashboard
The first ratings go to parents through the app, and the compliance dashboard goes on the staffroom screen so progress is visible. Mrs Lim means both as starting points. Parents read Developing as a fall, because last year’s results slips showed marks. She has not yet asked what a number on a screen does to the people under it.
Term 1, Week 4
Aisyah, in the corridor
Science rates against the end of the year and English against the current unit, so Aisyah is Developing in one and Secure in the other. By Friday Secondary 2 has sorted its subjects into “harsh” and “kind”. Nobody has said what the ratings are for, so students supply a meaning of their own. The scale was built to describe learning, and within a fortnight it is describing departments.
Term 1, Week 5
Mr Ravi, marking Secondary 2 science
Green-pen responses have settled into ritual: “I will use more evidence.” A few are written by a chatbot, which is a fair answer to a box that only checks something is there. Mr Ravi tries whole-class feedback instead, outside the policy. His marking halves and his lessons improve, and on the book-check checklist his books now look worse.
Term 1, Week 6
Mdm Rohani, at 3am on a break
The class parents’ chat group decides that without weighted assessments “nobody knows where our children stand”. Families sign up for tuition centres that set practice papers, so the school’s ratings now compete with a private set of scores. Mdm Rohani reads the thread the way the other parents do, as a statement of what counts. Aisyah has no tuition, and Mdm Rohani wonders whether she can afford to leave it that way.
Term 1, Week 8
Mr Ravi, the weekend after the book check
The first book check rates humanities “inconsistent”, and teachers start writing feedback for the book check rather than the student. Comments get longer, and one department loses a weekend to them. Mr Ravi watches colleagues write for a reader who checks format, not effect. The check was meant to observe feedback, and it has become its audience.
Term 1, Week 10
Mr Ravi and Mrs Lim, agreeing the probes
Staff agree three probes, including a ten-minute quiz each fortnight that is never recorded. Mr Ravi insists on the never. Anything written down will be looked at, and anything looked at becomes something to perform for. Mrs Lim writes that down and agrees, though she would like the data.
Term 2, Week 2
Aisyah, in the quiz
The quizzes find a misconception about energy that the checkpoint would not have shown until the end of the semester. Aisyah likes them because they do not count, and answers honestly, guesses included. Then her class asks for the quizzes to count. The measure works because it carries no stakes, and the students’ first instinct is to give it some.
Term 2, Week 4
Mrs Lim and Mdm Rohani, after the checkpoint
Aisyah freezes in the timed English paper and is rated Emerging, and Mdm Rohani is one of many parents who complain through the app. A teacher quietly rates borderline students up to head off the next round. Mrs Lim spots it only because the spread of ratings has a bump where the borderline should be. The ratings now measure learning, one timed hour, and how likely each family is to complain.
Term 2, Week 5
Mrs Lim, at the dashboard
The dashboard shows 96 per cent of books compliant. The target is nearly met and nobody in the school trusts it. Mrs Lim sees, late, that she was never reading the school from outside: the staffroom screen, the book-check checklist and the five-point scale were all moves, and the school answered each one.
Term 2, Week 5
Ms Tay and Mrs Lim, on the phone
The cluster newsletter cites the school as an exemplar of “feedback culture”, on the figure the school reported. Ms Tay rings to ask Mrs Lim to present at the next cluster meeting of principals. Mrs Lim tells her she does not believe the number. They both see the problem: with the school’s name now attached to 96 per cent, dropping the format check would read across the cluster as backsliding.
Term 2, Week 8
Mdm Rohani and Mr Ravi, on a Saturday morning
The Meet-the-Parents session has a Saturday morning slot after shift-working parents complained in Term 1, and Mdm Rohani comes for the first time in two years. The attendance record had her down as disengaged. It had been measuring the school’s timetable. Mr Ravi shows her Aisyah’s quiz record, the one measure nobody had a reason to bend.
Term 2, Week 10
Ms Tay, at her desk
Ms Tay traces where the twice-a-term ratings go once they reach her: the newsletter, a slide for principals, the targets other schools copy into their own plans. Her office asked for them because ratings were easy to collect. She proposes the cluster stop. Her manager asks what will replace them, and she has no number to offer, only a list of what the old ones were doing.
The semester let the school see its own instruments as moves. The ratings, the book-check checklist and the compliance dashboard each changed what students, teachers and parents did, and each then reported that change back as if it were news. The school changed what it could: the scale parents see was cut to three points, the quizzes stayed unrecorded, and Mrs Lim proposed dropping the format check. But the staff scale was not cut, the compliance figure still goes upward, the exemplar label still stands, and nobody yet knows what a three-point scale will teach parents to do.
Implications
The same five dimensions for all seven contributions. Read them across on the compare page.
From observer to participant. Most analysis assumes the person doing it stands outside the thing analysed, like a surveyor at the edge of a site. Second-order cybernetics rejected that position: the observer is part of the system observed (von Foerster, 2003). Stacey (2001) carried the point into management. Arthur (2015) carried it into economics, where people form expectations about outcomes that their expectations help create.
The sharpest form concerns measures. Campbell (1979) observed that the more a quantitative indicator is used for social decisions, the more it is corrupted and the more it distorts the process it was meant to monitor. Strathern (1997) gave the related Goodhart principle its popular form: when a measure becomes a target, it stops being a good measure. MacKenzie (2006) showed that financial models help produce the markets they describe. The pattern is the same each time. Framing, measurement and action enter the situation and change it.
Almost every other form of thinking places the thinker outside the object. Critical thinking assumes a stable object about which claims can be checked, so it has little to say about claims that change the object they describe. Systems thinking assumes a structure that holds still while it is mapped. Of the critiques of systems thinking, Stacey’s critique of the designer outside the system has the largest practical consequences. In a human system there is no position from which to map and redesign without being part of the pattern, and every intervention is reinterpreted by everyone else.
Reflexivity on its own does have a home. A negotiation between two parties is reflexive, and game theory handles it well. What no single discipline handles is reflexivity combined with emergence and adaptation, which is the ordinary condition of schools, markets and health systems. It is also the main reason concepts borrowed from physics fall short in human systems. Particles do not read the forecast.
It brings forward the effects that otherwise arrive later as “unintended consequences”: gaming, behavioural responses to a measure, and the loss of legitimacy when people feel watched rather than helped. Self-fulfilling and self-defeating prophecies (Merton, 1948), the gaming of targets and the performativity of models stop being anomalies to correct and become things to plan for.
It also makes the analyst’s own position an explicit variable. In the overlay it is a single check: how will the people and systems we are analysing respond to our framing, metrics and moves, and how will that change what we observe? Applied to a KPI dashboard, it replaces the assumption that indicators measure performance without changing it with a reflexivity review of each indicator, and with narratives collected alongside the numbers.
A working rule: any metric shown to users will change their behaviour. So before asking whether a number is accurate, ask what it will teach people to do once they can see it. That applies to dashboards on staffroom walls, to ratings sent home, to book-check checklists, and to the quizzes brought in to correct them.
It also asks leaders to accept that they are in the picture. A review is a move. So is a target, a pilot or a mention in a newsletter. Dashboards built for compliance reporting tend to flatten the weak signals that matter most, so a school needs other ways of sensing what is happening, and it needs to review its measures on a schedule, expecting each one to wear out. That is uncomfortable, and only partly in the school’s hands, because a school rarely controls the measures it reports upward.
The positionality paragraph. A note at the end of a report names the author’s standpoint, and nothing in the analysis changes because of it. Reflexivity becomes a confession instead of a variable.
The other hollow form is cynicism: every measure gets gamed, so measure nothing. That hands judgement to whoever is loudest, and it fails the test this strand sets itself, which asks complexity practice to be reflexive and testable. A measure-the-measure review earns its place when it ends in a change to an indicator or a decision, or in a stated reason for keeping both as they are.
Methods
The repertoire for this contribution, with an evidence rating for each method. Strong means tested in more than one field. Emerging means promising, and mostly case studies.
Assumption surfacing
ModerateWriting down what a plan takes for granted, including what each number is assumed to show, so the assumptions can be checked while the plan runs.
Measure-the-measure reviews
ModerateFor each indicator, asking what behaviour it rewards, who can move it without moving what it stands for, and what it will look like once people have learnt it.
Self-signified micro-narratives
ModeratePeople give short accounts of their experience and interpret them themselves, so the meaning is not assigned by the analyst. The accounts can still be performed for, like any regular survey.
Standpoint mapping
ModerateNaming whose view an analysis is built from, and whose it leaves out, so the analyst’s position is on the page and open to challenge.
Reading
Start with the first on the list.
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90.
The origin of Campbell’s law: indicators used for decisions come to distort what they monitor.
Strathern, M. (1997). ‘Improving ratings’: Audit in the British university system. European Review, 5(3), 305–321.
A critique of audit that gave the Goodhart principle its popular form.
Merton, R. K. (1948). The self-fulfilling prophecy. The Antioch Review, 8(2), 193–210.
The classic account of a prediction that brings about the outcome it predicts.
Stacey, R. D. (2001). Complex responsive processes in organizations: Learning and knowledge creation. Routledge.
The argument that managers cannot stand outside the organisation they are trying to change.
von Foerster, H. (2003). Understanding understanding: Essays on cybernetics and cognition. Springer.
Second-order cybernetics, and the observer as part of the system observed.
MacKenzie, D. (2006). An engine, not a camera: How financial models shape markets. MIT Press.
A detailed case of models helping to produce the world they claim only to describe.