Work / Note
A principle is not a finding
I set out to re-house a collection of sixty design principles and ended up checking their sources instead. Eleven of the sixty turned out not to be findings at all — and the document could not honestly be published until it said so on its own face.
The situation
Most applied fields carry an informal canon: a set of principles that circulate in decks, handbooks, and induction material, get cited in design reviews, and are treated by nearly everyone as settled. They are genuinely useful. They are also presented flat — numbered items in one list, in one voice, each carrying the same implied authority as the one above it.
I had a straightforward job with one such collection: take it and put it into a better form. A faithful conversion, nothing more. Checking the citations was meant to be the formality that came before the design work started.
The question
The formality did not survive the first afternoon. Several principles had citations that led nowhere in particular — to a book that does not contain the claim attributed to it, to a secondary source hedging in almost the same words as the other secondary source, or to no document at all. One widely taught rule turned out to be advice pointing in the opposite direction from the research group whose work it is named after.
Which raised the question the rest of the work turned on. If a collection sets a replicated experimental result and an undocumented rumour in the same type, at the same size, with the same confidence, what is it a collection of?
The honest answer is that it is a reading list wearing the clothes of a body of knowledge. And the difference matters most at exactly the moment the material is used: when someone has to defend a decision to a person who did not make it.
The move
Grade every item, on the face of the document, in the same position every time.
Four bands were enough. Something replicated: several peer-reviewed studies converging, with the direction of the effect stable. Something supported: a real primary source exists, but it is narrower in scope than the claim now made of it. A convention: documented provenance, no empirical test, and often an author who said plainly at the time that it was a rule of thumb. And contested: either the current evidence does not support the advice as written, or no primary source could be found at all.
The mark goes in the eyebrow of every item, not in an appendix and not in a footnote. An appendix is where you put a qualification you hope will not be read. If the grading is real it belongs where the claim is, at the same moment the reader meets it.
Where a mark is anything other than the top band, one line says why, in place, without hedging. That line is the expensive part to write and the whole reason the scheme is worth anything.
And the scheme has to be able to say something unflattering about the material it is attached to. A grading that only ever confirms is decoration with a vocabulary.
What it produced
Twenty-six of the sixty came through as replicated findings. Twenty-three rested on a source real but narrower than the claim it was carrying. Seven were craft conventions. Four were contested, two of them with no primary source of any kind. Six items ended up carrying a correction in place of the advice they had arrived with.
The first consequence was structural. The discard has to happen at the front, before any principle is taught, or the reader spends the whole document wondering what else has not been checked. Naming the four weakest items in the opening act is what buys the credibility of the other fifty-six. This is uncomfortable to do and it is the single highest-value move in the whole build.
The second was that labelling a convention turned out not to be a demotion. A heuristic that has organised layouts for two centuries has earned some deference; what it has not earned is a citation. Saying so lets people keep using it without having to pretend about where it came from, which is a better outcome than either defending it as research or throwing it out.
The third was that the marks changed how the material gets argued with. A flat list can only be accepted or ignored. A graded one can be disputed item by item — and a reader who wants to move something from convention to replicated now knows exactly what they would have to produce.
The point
What transfers
The reason this page exists, rather than the story that produced it.
- Any body of guidance that presents every item in one voice is making a claim about its own evidence that nobody has checked. That claim is usually the first thing to fail under inspection, and it takes the rest of the document down with it.
- Grade on the face of the material, not in an appendix. A qualification placed where it can be skipped has been designed to be skipped.
- A grading scheme that cannot produce an unflattering result is not a scheme. Build it so that it can demote your own favourite item, and then find out whether it does.
- Do the discard first. Naming what does not survive scrutiny, before teaching anything that does, is what makes the rest of the document believable — and it costs less than it feels like it will.
- Verification is design work, not a compliance step before design work. What the checking turns up usually changes what the thing needs to be.