Library / Glossary

The craft glossary

1002 terms for anyone who meets the vocabulary of educational technology: every term the craft guides define, in their words, and the wider jargon of each guide's field alongside.

The library

What each entry carries

A definition, in the words of the guide the term comes from.
The place in that guide where the term is referred to, so the definition can be read in context. A term from the guide’s field that the guide does not use is marked as such.
Two or three works to follow up, drawn from the guide's own references.

The terms come from Craft guides for educational technologists. For the other half of the vocabulary, see the context glossary.

Term	Definition	Referred to in	Read further
A
A/A test	A test in which both groups get the same version. It checks the system: any "significant" result is a false positive.	R13Experimentation and product analyticsNo peeking · slide 45 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
A/B test	An experiment in which users are randomly split between the current version (A) and a changed version (B), and the groups' outcomes are compared to estimate the effect of the change.	R13Experimentation and product analyticsFrom the field · not in the guide	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
A3 report	A one-page account of a problem, its analysis, the proposed countermeasures and the follow-up plan, named after the paper size and used at Toyota to structure problem-solving and mentoring.	R10Lean product developmentFrom the field · not in the guide	Shook, J. (2008). Managing to learn: Using the A3 management process to solve problems, gain agreement, mentor, and lead. Lean Enterprise Institute.
Abandonment	Stopping use of a technology after adopting it, often without anyone deciding to (Greenhalgh et al., 2017).	R19Adoption and change in schoolsPlan the exit · slide 49 (opens in a new tab)	Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017). Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. Journal of Medical Internet Research, 19(11), Article e367. doi Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Abductive reasoning	Reasoning from a set of observations to the most plausible explanation for them. It is the step by which synthesis moves from what was seen to a claim about why.	R06Design methods IFrom the field · not in the guide	Kolko, J. (2010). Abductive thinking and sensemaking: The drivers of design synthesis. Design Issues, 26(1), 15–28. doi
Absolute and relative risk	Absolute risk is the chance of an outcome. Relative risk compares two chances as a ratio, so 'twice as likely' can describe a rise from 1 to 2 in 1,000.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M., & Woloshin, S. (2007). Helping doctors and patients make sense of health statistics. Psychological Science in the Public Interest, 8(2), 53–96. doi Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican.
Academic analytics	Analytics applied at institution level to questions of management, such as enrolment, retention and resourcing, as distinct from analytics aimed at learning itself.	R15Learning analyticsFrom the field · not in the guide	Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5). er.educause.edu Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE Review, 42(4), 40–57.
Acceptance criteria	The conditions a piece of work must meet before the team and the user agree it is done.	R09Agile and Scrum in practiceShip small · slide 20 (opens in a new tab)	Jeffries, R. (2001). Essential XP: Card, conversation, confirmation. ronjeffries.com. ronjeffries.com Cohn, M. (2004). User stories applied: For agile software development. Addison-Wesley. mountaingoatsoftware.com
Accuracy	The share of all students a model is right about. It means little without the base rate: if one student in 20 will leave, a model that flags nobody is 95% accurate.	R15Learning analyticsThe base rate · slide 39 (opens in a new tab)	Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi
Acquiescence bias	The tendency of some respondents to agree with a statement whatever it says (Pew Research Center, 2021).	R05Evaluative researchQuestions that cannot lead · slide 31 (opens in a new tab)	Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press. archive.org Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. doi Pew Research Center. (2021). Writing survey questions. pewresearch.org
Action planning	Deciding in detail when, where and how a new practice will be used (Sims et al., 2021).	R21Professional learning that changes practiceThe second week · slide 47 (opens in a new tab)	Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov
Action research	Enquiry carried out by practitioners into their own practice, in cycles of planning, acting, observing and reflecting.	R21Professional learning that changes practiceFrom the field · not in the guide	Lewin, K. (1946). Action research and minority problems. Journal of Social Issues, 2(4), 34–46. doi
Actionable metric	A measure that ties a specific, repeatable action to an observed result, so that a change in it tells the team what to do next. The opposite of a vanity metric.	R10Lean product developmentFrom the field · not in the guide	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com Schonfeld, E. (2011, September 24). Founder stories: Eric Ries on "vanity metrics" and "success theater". TechCrunch. techcrunch.com
Activity data	Records a platform makes as a by-product of use: logins, clicks, page views, submissions and timestamps.	R15Learning analyticsThe misconception · slide 10 (opens in a new tab)	Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. doi Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68–84. doi Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R. S., & Hatala, M. (2015). Penetrating the black box of time-on-task estimation. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (pp. 184–193). ACM. doi
Adaptive learning	Software that changes the content, difficulty or order of tasks for each student according to a running estimate of what that student knows.	R16AI in learning productsFrom the field · not in the guide	U.S. Department of Education, Office of Educational Technology. (2023). Artificial intelligence and the future of teaching and learning: Insights and recommendations. files.eric.ed.gov OECD. (2026). OECD digital education outlook 2026: Exploring effective uses of generative AI in education. OECD Publishing. doi
Adjusted-Wald interval	A confidence interval for a proportion that behaves well with small samples (Sauro & Lewis, 2016).	R05Evaluative researchSmall and often · slide 26 (opens in a new tab)	Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann. shop.elsevier.com Agresti, A., & Coull, B. A. (1998). Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician, 52(2), 119–126. doi
ADKAR	A model of individual change with five steps in order: awareness, desire, knowledge, ability and reinforcement. It is used to find where a person or group is stuck.	R19Adoption and change in schoolsFrom the field · not in the guide	Hiatt, J. M. (2006). ADKAR: A model for change in business, government and our community. Prosci Learning Center Publications.
Administrative burden	The learning, compliance and psychological costs a system imposes on the people it deals with (Herd & Moynihan, 2018).	R02Service design and systems thinkingThe pushed-down work · slide 32 (opens in a new tab)	Herd, P., & Moynihan, D. P. (2018). Administrative burden: Policymaking by other means. Russell Sage Foundation. doi Moynihan, D., Herd, P., & Harvey, H. (2015). Administrative burden: Learning, psychological, and compliance costs in citizen-state interactions. Journal of Public Administration Research and Theory, 25(1), 43–69. doi
Adopter categories	Five groups sorted by how early they take up an innovation, from innovators to laggards (Rogers, 2003).	R19Adoption and change in schoolsThe reluctant adopter · slide 27 (opens in a new tab)	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press. Moore, G. A. (1991). Crossing the chasm: Marketing and selling technology products to mainstream customers. HarperBusiness.
Adoption	A lasting change in what people do because of a new tool or practice. More than access, and more than a login.	R19Adoption and change in schoolsThe spine · slide 13 (opens in a new tab)	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press. Cuban, L. (2001). Oversold and underused: Computers in the classroom. Harvard University Press. doi Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk
Aesthetic–usability effect	The tendency of people to judge a better-looking interface as easier to use, whether or not it is.	R08UX foundations and design systemsFrom the field · not in the guide	Tractinsky, N., Katz, A. S., & Ikar, D. (2000). What is beautiful is usable. Interacting with Computers, 13(2), 127–145. doi Lidwell, W., Holden, K., & Butler, J. (2010). Universal principles of design (Rev. and updated ed.). Rockport Publishers.
Affinity diagram	A wall of observations grouped by shared meaning, built from the bottom up, with each group labelled (Krause & Pernice, 2024).	R06Design methods ISynthesise together · slide 19 (opens in a new tab)	Beyer, H., & Holtzblatt, K. (1998). Contextual design: Defining customer-centered systems. Morgan Kaufmann. archive.org Scupin, R. (1997). The KJ method: A technique for analyzing data derived from Japanese ethnology. Human Organization, 56(2), 233–237. doi Krause, R., & Pernice, K. (2024). Affinity diagramming for collaboratively sorting UX findings and design ideas. Nielsen Norman Group. nngroup.com
Affordance	A relationship between the properties of an object and the abilities of the person using it, which determines how it could be used (Norman, 2013).	R08UX foundations and design systemsThe next step · slide 25 (opens in a new tab)	Norman, D. (2013). The design of everyday things (Rev. and expanded ed.). Basic Books. basicbooks.com Gibson, J. J. (1979). The ecological approach to visual perception. Houghton Mifflin. Norman, D. A. (1999). Affordance, conventions, and design. Interactions, 6(3), 38–43. doi
Age Appropriate Design Code	The United Kingdom regulator's statutory code of 15 standards for online services likely to be used by under-18s, including high privacy by default and no nudging to give up data. Also called the Children's Code.	R17Privacy, security and ethicsFrom the field · not in the guide	Information Commissioner's Office. (2020). Age appropriate design: A code of practice for online services. Information Commissioner's Office.
Agile Manifesto	A statement written in 2001 by 17 practitioners, setting out four values and 12 principles for software development; it prescribes no roles, events or tools (Beck et al., 2001).	R09Agile and Scrum in practiceThe misconception · slide 9 (opens in a new tab)	Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R. C., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D. (2001). Manifesto for Agile Software Development. agilemanifesto.org Fowler, M. (2018, August 25). The state of agile software in 2018 [Keynote transcript]. martinfowler.com. martinfowler.com
AI agent	A system in which a model plans and carries out a multi-step task by calling tools such as search, code or other software, with limited human direction.	R16AI in learning productsFrom the field · not in the guide	Russell, S., & Norvig, P. (2021). Artificial intelligence: A modern approach (4th ed.). Pearson. doi
AI literacy	The knowledge and skills that let people understand what AI systems do, use them well and judge their output critically.	R16AI in learning productsFrom the field · not in the guide	Miao, F., & Holmes, W. (2023). Guidance for generative AI in education and research. UNESCO. unesco.org Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–16. doi
AI Risk Management Framework	The US National Institute of Standards and Technology's framework, which organises AI risk work into four functions: govern, map, measure and manage (National Institute of Standards and Technology, 2023).	R16AI in learning productsEvaluate first · slide 19 (opens in a new tab)	National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). U.S. Department of Commerce. doi National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. doi
AI-writing detector	A tool that labels text as human- or AI-written; seven detectors wrongly flagged most essays by non-native English writers, so a flag is a reason for a conversation, never proof (Liang et al., 2023).	R16AI in learning productsEvaluate first · slide 20 (opens in a new tab)	Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7), Article 100779. doi
Alert threshold	The score above which a student is flagged. Moving it changes how many students are flagged and how many flags are wrong.	R15Learning analyticsShow the uncertainty · slide 32 (opens in a new tab)	Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi
Algorithmic bias	Systematic differences in how well an automated system works for different groups of people, such as a marking model that is less accurate for some students than others.	R16AI in learning productsFrom the field · not in the guide	National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). U.S. Department of Commerce. doi Baker, R. S., & Hawn, A. (2022). Algorithmic bias in education. International Journal of Artificial Intelligence in Education, 32(4), 1052–1092. doi
Alignment diagram	A picture, such as a journey map, service blueprint, experience map or ecosystem model, that helps an organisation see the user's experience alongside its own work (Kalbach, 2020).	R02Service design and systems thinkingThe spine · slide 13 (opens in a new tab)	Kalbach, J. (2020). Mapping experiences: A complete guide to customer alignment through journeys, blueprints, and diagrams (2nd ed.). O'Reilly Media. openlibrary.org
Alpha phase	The delivery phase in which a team tries out different solutions to the problems found in discovery, tests its riskiest assumptions and expects to throw away the code (Government Digital Service, 2019).	R10Lean product developmentThe riskiest assumption · slide 20 (opens in a new tab)	Government Digital Service. (2019). How the alpha phase works. GOV.UK Service Manual. gov.uk
Alternative text (alt text)	A short written description attached to an image, read aloud by screen readers and shown when the image fails to load.	R08UX foundations and design systemsFrom the field · not in the guide	World Wide Web Consortium. (2023). Web Content Accessibility Guidelines (WCAG) 2.2 (W3C Recommendation). w3.org Horton, S., & Quesenbery, W. (2013). A web for everyone: Designing accessible user experiences. Rosenfeld Media.
Always-valid p-value	A p-value designed to stay correct however often a running test is checked (Johari et al., 2022).	R13Experimentation and product analyticsNo peeking · slide 45 (opens in a new tab)	Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2022). Always valid inference: Continuous monitoring of A/B tests. Operations Research, 70(3), 1806–1821. doi
Analysis of variance (ANOVA)	A test of whether the means of three or more groups differ, made by comparing the variation between groups with the variation inside them.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Fisher, R. A. (1925). Statistical methods for research workers. Oliver and Boyd. Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE.
Analytic memo	A note written by the analyst during coding that records an emerging idea, a question or the reason for a decision, and that later feeds the write-up.	R06Design methods IFrom the field · not in the guide	Saldaña, J. (2025). The coding manual for qualitative researchers (5th ed.). SAGE.
Andragogy	The theory and practice of teaching adults, which assumes that they are self-directed, bring experience, and learn best what they can use soon.	R21Professional learning that changes practiceFrom the field · not in the guide	Knowles, M. S. (1973). The adult learner: A neglected species. Gulf Publishing.
Anonymisation	Altering data so that no person can be identified from it by any means reasonably likely to be used. Truly anonymised data falls outside data protection law.	R17Privacy, security and ethicsFrom the field · not in the guide	Article 29 Data Protection Working Party. (2014). Opinion 05/2014 on anonymisation techniques (WP 216). European Commission. Ohm, P. (2010). Broken promises of privacy: Responding to the surprising failure of anonymization. UCLA Law Review, 57(6), 1701–1777.
API	Application programming interface: the published set of requests a system will accept from other software.	R11Software development for non-engineersThe spine · slide 15 (opens in a new tab)	MDN Web Docs. (n.d.). How the web works. Mozilla. Retrieved September 17, 2026, from developer.mozilla.org
Area under the curve (AUC)	A score from 0.5 (chance) to 1 (perfect) for how well a predictive model ranks students who had the outcome above those who did not, across all thresholds.	R15Learning analyticsFrom the field · not in the guide	Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. doi
Artefact	An object that people make or use in their work, such as a mark book, planner or annotated worksheet, collected or photographed as evidence of how the work is really done.	R04Discovery researchFrom the field · not in the guide	Beyer, H., & Holtzblatt, K. (1998). Contextual design: Defining customer-centered systems. Morgan Kaufmann. archive.org
Assent	A child's own agreement to take part in research, sought alongside a guardian's consent (British Educational Research Association, 2024). R05 A child's own agreement to take part, sought alongside consent from a responsible adult (British Educational Research Association, 2024). R13 Agreement to take part given by children too young to consent; principles of consent apply to children and young people as well as to adults (British Educational Research Association, 2024).	R04Discovery researchTwo yeses · slide 43 (opens in a new tab) R05Evaluative researchThe real pace, the real room · slide 40 (opens in a new tab) R13Experimentation and product analyticsWrite it down first · slide 34 (opens in a new tab)	British Educational Research Association. (2024). Ethical guidelines for educational research (5th ed.). bera.ac.uk Lundy, L. (2007). ‘Voice’ is not enough: Conceptualising Article 12 of the United Nations Convention on the Rights of the Child. British Educational Research Journal, 33(6), 927–942. doi United Nations. (1989). Convention on the Rights of the Child (General Assembly resolution 44/25). ohchr.org
Assisted digital	Support given to people who cannot use an online service unaided, by phone, in person or through someone acting for them, so that they still get the same result.	R02Service design and systems thinkingFrom the field · not in the guide	Government Digital Service. (2019). Service Standard. GOV.UK. gov.uk
Assistive technology	Hardware or software that people with disabilities use to operate a computer or read its content, such as screen readers, screen magnifiers, switch devices and voice control.	R08UX foundations and design systemsFrom the field · not in the guide	World Wide Web Consortium. (2023). Web Content Accessibility Guidelines (WCAG) 2.2 (W3C Recommendation). w3.org Horton, S., & Quesenbery, W. (2013). A web for everyone: Designing accessible user experiences. Rosenfeld Media.
Assumption	A belief a plan depends on that has not yet been shown to be true.	R10Lean product developmentThe spine · slide 12 (opens in a new tab)	Bland, D. J., & Osterwalder, A. (2019). Testing business ideas: A field guide for rapid experimentation. Wiley. books.google.com Gothelf, J., & Seiden, J. (2013). Lean UX: Applying lean principles to improve user experience. O'Reilly Media. oreilly.com
Assumptions mapping	Sorting a plan's beliefs by how important they are and how much evidence supports them (Bland, 2020).	R10Lean product developmentThe riskiest assumption · slide 19 (opens in a new tab)	Bland, D. J. (2020, August 4). How assumptions mapping can focus your teams on running experiments that matter. Strategyzer. strategyzer.com Bland, D. J., & Osterwalder, A. (2019). Testing business ideas: A field guide for rapid experimentation. Wiley. books.google.com
At-risk student	A student a model or a teacher judges likely to fail, fall behind or leave without extra support. The label is a prediction, not a fact about the student.	R15Learning analyticsFrom the field · not in the guide	Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267–270). ACM. doi Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z., Vaclavek, J., & Wolff, A. (2015). OU Analyse: Analysing at-risk students at The Open University. Learning Analytics Review, LAK15-1, 1–16. oro.open.ac.uk
Atomic design	A mental model of interfaces as built from atoms, molecules, organisms, templates and pages, which treats a screen as both a whole and a set of shared parts (Frost, 2016).	R08UX foundations and design systemsSystem first · slide 31 (opens in a new tab)	Frost, B. (2016). Atomic design. Brad Frost Web. atomicdesign.bradfrost.com
Atomic research	Breaking research into small linked units, such as facts, insights and recommendations, that can be found and reused (Pidcock, 2018).	R06Design methods IKeep the trail · slide 26 (opens in a new tab)	Pidcock, D. (2018). What is atomic UX research? Prototypr. blog.prototypr.io Sharon, T. (2016). The atomic unit of a research insight. Medium. tsharon.medium.com Sharon, T. (2018). Foundations of atomic research. Medium. tsharon.medium.com
Attack surface	Every point at which an attacker could try to get into or take data out of a system: login pages, APIs, integrations, devices and the people who use them.	R17Privacy, security and ethicsFrom the field · not in the guide	Manadhata, P. K., & Wing, J. M. (2011). An attack surface metric. IEEE Transactions on Software Engineering, 37(3), 371–386. doi Shostack, A. (2014). Threat modeling: Designing for security. Wiley.
Attention budget	This guide's name for the limited working memory a user has, shared between the task and the interface.	R08UX foundations and design systemsThe spine · slide 13 (opens in a new tab)	Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31(2), 261–292. doi Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. doi Mayer, R. E. (2024). The past, present, and future of the cognitive theory of multimedia learning. Educational Psychology Review, 36(1), Article 8. doi
Attitudinal method	A method that records what people say: their views, ratings and feelings.	R05Evaluative researchThe spine · slide 13 (opens in a new tab)	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com
Attitudinal research	Research that collects what people say: their beliefs, opinions and reports (Rohrer, 2022).	R04Discovery researchThe misconception · slide 8 (opens in a new tab)	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com LaPiere, R. T. (1934). Attitudes vs. actions. Social Forces, 13(2), 230–237. doi
Attrition	The loss of participants from a study before outcomes are measured. Differential attrition, where one group loses more than the other, can bias the estimated effect.	R13Experimentation and product analyticsFrom the field · not in the guide	What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0 (WWC 2022008). U.S. Department of Education, Institute of Education Sciences. ies.ed.gov Education Endowment Foundation. (2022). Statistical analysis guidance for EEF evaluations. d2tic4wvo1iusb.cloudfront.net
Audience segmentation	Dividing the people a change affects into groups by role, need or situation, so that each group gets the message that applies to it.	R20Communicating change to schoolsFrom the field · not in the guide	Slater, M. D. (1996). Theory and method in health audience segmentation. Journal of Health Communication, 1(3), 267–284. doi
Audience-first message	A message that opens with what changes for the reader, from when, and what they must do.	R20Communicating change to schoolsLead with the teacher · slide 19 (opens in a new tab)	Government Digital Service. (n.d.-e). Understand content design. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk Government Digital Service. (n.d.-a). Write change notes. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk
Audit log	A protected record of who did what in a system and when, such as who viewed or exported a student's record, used to detect misuse and investigate incidents.	R17Privacy, security and ethicsFrom the field · not in the guide	Kent, K., & Souppaya, M. (2006). Guide to computer security log management (NIST Special Publication 800-92). National Institute of Standards and Technology. doi
Austin's Butterfly	An EL Education film in which first-grade students use critique to help a classmate take a scientific drawing of a butterfly through several drafts (Berger et al., 2014).	R07Design methods IICritique against the goal · slide 33 (opens in a new tab)	Berger, R., Rugen, L., & Woodfin, L. (2014). Leaders of their own learning: Transforming schools through student-engaged assessment. Jossey-Bass. wiley.com Berger, R. (2003). An ethic of excellence: Building a culture of craftsmanship with students. Heinemann. heinemann.com
Authentication and authorisation	Authentication proves who a user is; authorisation decides what that user may see or do. Many breaches of student data come from a failure of the second.	R17Privacy, security and ethicsFrom the field · not in the guide	Grassi, P. A., Garcia, M. E., & Fenton, J. L. (2017). Digital identity guidelines (NIST Special Publication 800-63-3). National Institute of Standards and Technology. doi OWASP Foundation. (2021). OWASP Top 10:2021. OWASP Foundation.
Automated decision-making	A decision about a person made by a system with no meaningful human involvement, such as automatic grading or placement. Several laws give a right to human review.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Wachter, S., Mittelstadt, B., & Floridi, L. (2017). Why a right to explanation of automated decision-making does not exist in the General Data Protection Regulation. International Data Privacy Law, 7(2), 76–99. doi
Automated essay scoring	Software that assigns a mark to a piece of extended writing, using a model trained on essays that people have already marked.	R16AI in learning productsFrom the field · not in the guide	Shermis, M. D., & Burstein, J. (Eds.). (2013). Handbook of automated essay evaluation: Current applications and new directions. Routledge.
Automation bias	The tendency to accept an automated system's output without checking it, even when other evidence contradicts it. It weakens human review of AI decisions.	R16AI in learning productsFrom the field · not in the guide	Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. doi
B
Back end	The servers and databases that hold a system's rules and records, out of the user's sight.	R11Software development for non-engineersThe spine · slide 15 (opens in a new tab)	MDN Web Docs. (n.d.). How the web works. Mozilla. Retrieved September 17, 2026, from developer.mozilla.org
Backlog	The ranked list of work a team may do next. In Scrum, the Product Backlog carries a Product Goal (Schwaber & Sutherland, 2020).	R09Agile and Scrum in practiceThe spine · slide 16 (opens in a new tab)	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Backlog refinement	The continuing work of splitting, clarifying, sizing and reordering Product Backlog items so that the ones near the top are ready to be worked on.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org Rubin, K. S. (2012). Essential Scrum: A practical guide to the most popular agile process. Addison-Wesley.
Backstage	Work that supports a service but that the user does not see.	R02Service design and systems thinkingMap the backstage · slide 19 (opens in a new tab)	Shostack, G. L. (1984). Designing services that deliver. Harvard Business Review, 62(1), 133–139. hbr.org Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008). Service blueprinting: A practical technique for service innovation. California Management Review, 50(3), 66–94. doi Gibbons, S. (2017, August 27). Service blueprints: Definition. Nielsen Norman Group. nngroup.com
Backward planning	Planning professional development from the student learning you want back to what teachers would do differently, what the school must provide, what teachers must learn, and only then the session (Guskey, 2002).	R21Professional learning that changes practiceMeasure practice · slide 42 (opens in a new tab)	Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45–51. ascd.org
Balanced design	A programme with at least one mechanism for each of the four purposes: insight, goals, techniques and embedding practice (Sims et al., 2021).	R21Professional learning that changes practiceThe second week · slide 48 (opens in a new tab)	Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov
Balancing loop	A feedback loop that works against change and pulls a system back towards a goal or limit, as a thermostat does.	R02Service design and systems thinkingFrom the field · not in the guide	Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green. Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. Doubleday/Currency.
Base rate	How common an outcome is in a group before any prediction is made. R18 How common an outcome is in the population before any test or model is applied.	R15Learning analyticsThe base rate · slide 38 (opens in a new tab) R18Basic statistics for educational technologistsRisk and base rates · slide 43 (opens in a new tab)	Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M., & Woloshin, S. (2007). Helping doctors and patients make sense of health statistics. Psychological Science in the Public Interest, 8(2), 53–96. doi Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin, 52(3), 194–216. doi
Baseline equivalence	Evidence that the intervention and comparison groups were similar on key characteristics, such as prior attainment, before the intervention began.	R13Experimentation and product analyticsFrom the field · not in the guide	What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0 (WWC 2022008). U.S. Department of Education, Institute of Education Sciences. ies.ed.gov
Batch	A set of changes released together. Larger batches delay feedback on every change inside them (Reinertsen, 2009).	R09Agile and Scrum in practiceThe spine · slide 14 (opens in a new tab)	Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com
Bayes' theorem	The rule for updating the probability of something in the light of new evidence, combining how likely it was beforehand with how well the evidence fits.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. doi Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian data analysis (3rd ed.). CRC Press. doi
Before-and-after comparison	Comparing a measure before and after a change. It credits the change with everything else that happened in between, such as a new term, an exam season or a new timetable.	R13Experimentation and product analyticsThe misconception · slide 8 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
Behavioural method	A method that records what people do: their actions, successes and errors.	R05Evaluative researchThe spine · slide 13 (opens in a new tab)	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com
Behavioural research	Research that collects what people do, by watching or measuring it (Rohrer, 2022).	R04Discovery researchThe misconception · slide 8 (opens in a new tab)	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com Nielsen, J. (2001). First rule of usability? Don't listen to users. Nielsen Norman Group. nngroup.com
Behavioural variable	A dimension on which the people studied differ in what they do, such as how often they mark work online. Personas are built from clusters of people who sit together on several.	R06Design methods IFrom the field · not in the guide	Laubheimer, P. (2020). 3 persona types: Lightweight, qualitative, and statistical. Nielsen Norman Group. nngroup.com Cooper, A., Reimann, R., Cronin, D., & Noessel, C. (2014). About face: The essentials of interaction design (4th ed.). Wiley.
Benchmark	A public, standard test of a model's general ability. Useful for comparing models, weak evidence about a specific classroom task.	R16AI in learning productsEvaluate first · slide 19 (opens in a new tab)	National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. doi
Best Evidence Synthesis	New Zealand's synthesis on teacher professional learning, drawing on 97 studies of professional development that improved outcomes for students (Timperley et al., 2007).	R21Professional learning that changes practiceThe spine · slide 16 (opens in a new tab)	Timperley, H., Wilson, A., Barrar, H., & Fung, I. (2007). Teacher professional learning and development: Best evidence synthesis iteration. Ministry of Education, New Zealand. thehub.sia.govt.nz Timperley, H. (2008). Teacher professional learning and development (Educational Practices Series No. 18). International Academy of Education; International Bureau of Education. iaoed.org
Bet	A decision to spend effort on the expectation that it will change someone's behaviour in a useful way. It can lose.	R01Digital product managementThe spine · slide 12 (opens in a new tab)	Seiden, J. (2019). Outcomes over output: Why customer behavior is the key metric for business success. Sense & Respond Press. Kohavi, R., Crook, T., & Longbotham, R. (2009). Online experimentation at Microsoft [Paper presentation]. Third Workshop on Data Mining Case Studies and Practice Prize. exp-platform.com
Beta phase	The stage of a public digital service in which a working version is built and opened to real users, first by invitation and then publicly, while it is still being improved.	R01Digital product managementFrom the field · not in the guide	Government Digital Service. (2019). Service Standard. GOV.UK. gov.uk
Bias audit	A test, before launch and at intervals after, of whether a model's accuracy and errors differ between groups of students, such as by sex, ethnicity, language or disability.	R17Privacy, security and ethicsFrom the field · not in the guide	Baker, R. S., & Hawn, A. (2022). Algorithmic bias in education. International Journal of Artificial Intelligence in Education, 32(4), 1052–1092. doi Barocas, S., Hardt, M., & Narayanan, A. (2023). Fairness and machine learning: Limitations and opportunities. MIT Press.
Biometric data	Data from a person's body or behaviour that can identify them, such as a face image, fingerprint or voice print, used in some attendance, payment and proctoring systems.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 77–91.
Black-box monitoring	Checking a live system from the outside, as a user would, for example with an automated login.	R11Software development for non-engineersKnowing it is broken · slide 39 (opens in a new tab)	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Blameless postmortem	A review of an incident that looks for contributing causes without blaming individuals (Beyer et al., 2016).	R11Software development for non-engineersKnowing it is broken · slide 39 (opens in a new tab)	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Blameless review	A review of an incident that looks for contributing causes without blaming a person or team (Lunney & Lueder, 2016).	R22Support and feedback loopsTime to fix · slide 41 (opens in a new tab)	Lunney, J., & Lueder, S. (2016). Postmortem culture: Learning from failure. In B. Beyer, C. Jones, J. Petoff, & N. R. Murphy (Eds.), Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Blended learning	A course design that combines face-to-face sessions with online study, each used for what it does best.	R21Professional learning that changes practiceFrom the field · not in the guide	Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7(2), 95–105. doi
Blue-green deployment	Running two identical production environments and switching traffic between them, so a release can be reversed quickly (Fowler, 2010).	R12DevOps and continuous deliverySmall and reversible · slide 26 (opens in a new tab)	Fowler, M. (2010). Blue green deployment. martinfowler.com. martinfowler.com Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Bodystorming	Generating and trying out ideas by acting them out physically, ideally in the place where the product will be used.	R07Design methods IIFrom the field · not in the guide	Oulasvirta, A., Kurvinen, E., & Kankainen, T. (2003). Understanding contexts by being there: Case studies in bodystorming. Personal and Ubiquitous Computing, 7(2), 125–134. doi
Brainstorming	Group idea generation under four rules: go for quantity, hold back criticism, welcome unusual ideas, and build on the ideas of others.	R03Design thinking, used honestlyFrom the field · not in the guide	Diehl, M., & Stroebe, W. (1987). Productivity loss in brainstorming groups: Toward the solution of a riddle. Journal of Personality and Social Psychology, 53(3), 497–509. doi Osborn, A. F. (1953). Applied imagination: Principles and procedures of creative thinking. Scribner.
Brainwriting	Idea generation in which people write their ideas down in silence and pass them round for others to build on, which avoids waiting for a turn to speak.	R03Design thinking, used honestlyFrom the field · not in the guide	Diehl, M., & Stroebe, W. (1987). Productivity loss in brainstorming groups: Toward the solution of a riddle. Journal of Personality and Social Psychology, 53(3), 497–509. doi Paulus, P. B., & Yang, H.-C. (2000). Idea generation in groups: A basis for creativity in organizations. Organizational Behavior and Human Decision Processes, 82(1), 76–87. doi
Branch	A separate line of work in version control, where changes can be made and tested without affecting the main line until they are merged back in.	R11Software development for non-engineersFrom the field · not in the guide	Loeliger, J., & McCullough, M. (2012). Version control with Git: Powerful tools and techniques for collaborative software development (2nd ed.). O'Reilly Media.
Breach notification	The legal duty to tell the regulator, and often the people affected, about a personal data breach within a set time: 72 hours under the GDPR, three calendar days in Singapore.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Personal Data Protection Act 2012 (2020 Rev. Ed.) (Singapore).
Breaking change	A release that stops existing uses of a shared part working as before, marked by a new major version number.	R08UX foundations and design systemsRun it as a product · slide 44 (opens in a new tab)	Preston-Werner, T. (n.d.). Semantic versioning 2.0.0. Retrieved September 17, 2026, from semver.org Government Digital Service. (n.d.-c). Versioning [GOV.UK Frontend documentation]. GitHub. Retrieved September 17, 2026, from github.com
Brooks's law	Adding people to a late software project makes it later, because new people must be trained and must coordinate with everyone else (Brooks, 1975).	R11Software development for non-engineersReading a proposal · slide 44 (opens in a new tab)	Brooks, F. P., Jr. (1975). The mythical man-month: Essays on software engineering. Addison-Wesley. Anniversary edition published 1995. informit.com
Bug report	A report of a software fault that gives the steps to reproduce it, what was expected, what happened, and the device and version in use.	R22Support and feedback loopsFrom the field · not in the guide
Build artefact	The packaged, versioned output of a build. The pipeline creates it once and deploys that same package to every environment, so what reaches users is exactly what was tested.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Build trap	Measuring success by outputs rather than outcomes (Perri, 2018).	R01Digital product managementThe misconception · slide 8 (opens in a new tab)	Perri, M. (2018). Escaping the build trap: How effective product management creates real value. O'Reilly Media. Seiden, J. (2019). Outcomes over output: Why customer behavior is the key metric for business success. Sense & Respond Press.
Build-measure-learn	The loop of turning an idea into something testable, measuring the response and deciding what to do next (Ries, 2011).	R10Lean product developmentThe spine · slide 13 (opens in a new tab)	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com Blank, S. (2013). Why the lean start-up changes everything. Harvard Business Review, 91(5), 63–72. hbr.org
Burndown chart	A chart of the work remaining in a sprint or release against time, used to see whether the team is on course to finish.	R09Agile and Scrum in practiceFrom the field · not in the guide	Cohn, M. (2005). Agile estimating and planning. Prentice Hall.
Business glossary	An agreed list of an organisation's terms, such as "persistent absence" or "enrolled", each with one definition and a named owner.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Business intelligence (BI)	The tools and practices for turning an organisation's data into reports, dashboards and queries that managers use to monitor performance and make decisions.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Business model canvas	A one-page chart of nine blocks, including customer segments, value proposition, channels, revenue and costs, used to describe how an organisation creates and captures value and to list the assumptions behind it.	R10Lean product developmentFrom the field · not in the guide	Blank, S. (2013). Why the lean start-up changes everything. Harvard Business Review, 91(5), 63–72. hbr.org Osterwalder, A., & Pigneur, Y. (2010). Business model generation: A handbook for visionaries, game changers, and challengers. Wiley.
Buy-in	Real agreement, from the people who must carry out a change, that it is worth doing. It is distinct from compliance.	R19Adoption and change in schoolsFrom the field · not in the guide	Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk Kotter, J. P., & Whitehead, L. A. (2010). Buy-in: Saving your good idea from getting shot down. Harvard Business Review Press.
C
Cache	A store of recently fetched or computed results kept close to where they are needed, so that repeat requests are answered faster and put less load on the system behind it.	R11Software development for non-engineersFrom the field · not in the guide	Fielding, R., Nottingham, M., & Reschke, J. (Eds.). (2022). HTTP caching (RFC 9111). Internet Engineering Task Force. doi
Caliper Analytics	A standard that describes learning activity, what learners did, in a common form (1EdTech Consortium, n.d.-a).	R14Data strategy and governanceIdentifiers first · slide 39 (opens in a new tab)	1EdTech Consortium. (n.d.-a). Caliper Analytics. Retrieved September 17, 2026, from 1edtech.org
Call to action	The line in a message that tells the reader exactly what to do next, such as a button, a link or a dated instruction.	R20Communicating change to schoolsFrom the field · not in the guide
Campbell’s law	The more any quantitative social indicator is used for social decision-making, the more subject it is to corruption pressures, and the more it distorts the social processes it is meant to monitor (Campbell, 1979).	R15Learning analyticsActivity is not learning · slide 21 (opens in a new tab)	Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. doi
Canary release	Rolling a change out to a small group of users first, and widening it only if the signals hold (Sato, 2014).	R12DevOps and continuous deliverySmall and reversible · slide 26 (opens in a new tab)	Sato, D. (2014). Canary release. martinfowler.com. martinfowler.com Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Canned response	A pre-written reply to a common question that support staff insert into a ticket and adapt. Also called a macro.	R22Support and feedback loopsFrom the field · not in the guide
Capacity planning	Forecasting the demand a system will face, including its predictable peaks, and making sure enough computing resource is in place, tested and paid for before that demand arrives.	R11Software development for non-engineersFrom the field · not in the guide	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google Government Digital Service. (2017, February 6). Test your service's performance. GOV.UK Service Manual. gov.uk
Card sorting	A method in which participants group labelled cards in a way that makes sense to them, used to learn how users expect content to be organised and named.	R05Evaluative researchFrom the field · not in the guide	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com Spencer, D. (2009). Card sorting: Designing usable categories. Rosenfeld Media.
Cargo-cult agile	Copying the visible rituals of agile, such as sprints and stand-ups, without shortening the feedback loop they were meant to serve.	R09Agile and Scrum in practiceThe misconception · slide 8 (opens in a new tab)	Fowler, M. (2018, August 25). The state of agile software in 2018 [Keynote transcript]. martinfowler.com. martinfowler.com Thomas, D. (2014, March 4). Agile is dead (long live agility). pragdave.me. pragdave.me
Cascade	The chain of people a change message passes through, from the centre to the classroom.	R20Communicating change to schoolsThe spine · slide 14 (opens in a new tab)	Spillane, J. P., Reiser, B. J., & Reimer, T. (2002). Policy implementation and cognition: Reframing and refocusing implementation research. Review of Educational Research, 72(3), 387–431. doi Coburn, C. E. (2001). Collective sensemaking about reading: How teachers mediate reading policy in their professional communities. Educational Evaluation and Policy Analysis, 23(2), 145–170. doi Coburn, C. E. (2005). Shaping teacher sensemaking: School leaders and the enactment of reading policy. Educational Policy, 19(3), 476–509. doi
Causal loop diagram	A diagram of the variables in a system joined by arrows marked to show whether each one pushes the next up or down, used to find its feedback loops.	R02Service design and systems thinkingFrom the field · not in the guide	Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. Doubleday/Currency. Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. McGraw-Hill.
Central model	A model in which one team collects, stores and serves all data. It is simple to govern and often slow, because everything waits on the central team.	R14Data strategy and governanceNamed owners · slide 26 (opens in a new tab)	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Champion	A teacher or staff member who knows a tool well, helps colleagues with it and passes problems on. Also called a super-user.	R22Support and feedback loopsChampions · slide 45 (opens in a new tab)	Yuan, C. T., Bradley, E. H., & Nembhard, I. M. (2015). A mixed methods study of how clinician 'super users' influence others during the implementation of electronic health records. BMC Medical Informatics and Decision Making, 15, Article 26. doi
Change advisory board	A group that reviews and approves changes before they go live. Often called a CAB.	R12DevOps and continuous deliveryFreeze by exception · slide 43 (opens in a new tab)	Forsgren, N., Smith, D., Humble, J., & Frazelle, J. (2019). Accelerate: State of DevOps 2019. DORA and Google Cloud. dora.dev Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com
Change agent	A person who works to bring about the adoption of an innovation in a group, often from outside it, on behalf of the organisation promoting the change.	R19Adoption and change in schoolsFrom the field · not in the guide	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Change curve	A picture of how morale and performance often dip after a change is announced and recover as people accept it, adapted from Kübler-Ross's stages of grief.	R19Adoption and change in schoolsFrom the field · not in the guide	Kübler-Ross, E. (1969). On death and dying. Macmillan.
Change fail rate	The share of deployments that need immediate intervention afterwards (DORA, 2026).	R12DevOps and continuous deliveryThe dora measures · slide 37 (opens in a new tab)	DORA. (2026). DORA's software delivery performance metrics. Google Cloud. dora.dev Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com Harvey, N. (2026). A history of DORA's software delivery metrics. DORA. dora.dev
Change fatigue	Weariness and disengagement in staff who have been asked to absorb too many changes in too short a time.	R19Adoption and change in schoolsFrom the field · not in the guide	Bernerth, J. B., Walker, H. J., & Harris, S. G. (2011). Change fatigue: Development and initial validation of a new measure. Work & Stress, 25(4), 321–337. doi
Change freeze	A period in which no changes may be deployed, except agreed urgent fixes.	R12DevOps and continuous deliveryFreeze by exception · slide 42 (opens in a new tab)	Majors, C. (2019). Friday deploy freezes are exactly like murdering puppies. charity.wtf. charity.wtf Majors, C. (2025). On Friday deploys: Sometimes that puppy needs murdering. charity.wtf. charity.wtf
Change impact assessment	An analysis of which groups a change affects, in what way and how much, used to decide who needs which message, training and support.	R20Communicating change to schoolsFrom the field · not in the guide
Change lead time	The time from a change being committed to version control to it running in production (DORA, 2026).	R12DevOps and continuous deliveryThe dora measures · slide 37 (opens in a new tab)	DORA. (2026). DORA's software delivery performance metrics. Google Cloud. dora.dev Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com Harvey, N. (2026). A history of DORA's software delivery metrics. DORA. dora.dev
Change management	The planned work of moving people and an organisation from one way of working to another: preparing, supporting and following up, not only announcing.	R19Adoption and change in schoolsFrom the field · not in the guide	Fullan, M. (2001). Leading in a culture of change. Jossey-Bass. Kotter, J. P. (1996). Leading change. Harvard Business School Press. doi
Change note	A short public record of what changed on a page or in a product, written so a reader can act on it (Government Digital Service, n.d.-a).	R20Communicating change to schoolsWhat was removed · slide 39 (opens in a new tab)	Government Digital Service. (n.d.-a). Write change notes. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk Government Digital Service. (n.d.-b). Help users prepare for change. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk
Changelog	A curated, dated list of the notable changes in each version of a product (Lacan, 2026).	R20Communicating change to schoolsWhat was removed · slide 38 (opens in a new tab)	Lacan, O. (2026). Keep a Changelog (Version 2.0.0). keepachangelog.com
Channel	A medium through which a user deals with a service, such as a website, an app, a phone line, a letter or a school office counter.	R02Service design and systems thinkingFrom the field · not in the guide	Polaine, A., Løvlie, L., & Reason, B. (2013). Service design: From insight to implementation. Rosenfeld Media. Stickdorn, M., Hormess, M. E., Lawrence, A., & Schneider, J. (2018). This is service design doing: Applying service design thinking in the real world. O'Reilly Media.
Chaos engineering	Deliberately injecting failures, such as shutting down a server, into a system under controlled conditions to check that it copes as designed before a real failure tests it.	R12DevOps and continuous deliveryFrom the field · not in the guide	Basiri, A., Behnam, N., de Rooij, R., Hochstein, L., Kosewski, L., Reynolds, J., & Rosenthal, C. (2016). Chaos engineering. IEEE Software, 33(3), 35–41. doi
Children's Online Privacy Protection Act (COPPA)	The 1998 United States law that requires online services directed at children under 13, or knowingly collecting their data, to obtain verifiable parental consent first.	R17Privacy, security and ethicsFrom the field · not in the guide	Children's Online Privacy Protection Act of 1998, 15 U.S.C. §§ 6501–6506. Children's Online Privacy Protection Rule, 16 C.F.R. Part 312.
Children's rights by design	Designing digital products so that they respect children's rights to privacy, protection, participation, play and education from the outset.	R17Privacy, security and ethicsFrom the field · not in the guide	United Nations Committee on the Rights of the Child. (2021). General comment No. 25 (2021) on children's rights in relation to the digital environment (CRC/C/GC/25). United Nations. United Nations. (1989). Convention on the Rights of the Child. United Nations.
Chunking	Grouping separate items into a few meaningful units, as with the blocks of a phone number, so that more can be held in working memory.	R08UX foundations and design systemsFrom the field · not in the guide	Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81–97. doi Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24(1), 87–114. doi
Churn rate	The share of users or customers who stop using a product in a given period. Its complement is the retention rate.	R01Digital product managementFrom the field · not in the guide	Croll, A., & Yoskovitz, B. (2013). Lean analytics: Use data to build a better startup faster. O'Reilly Media.
Circular	An official notice sent by a ministry or authority to all the schools or bodies it oversees, usually numbered and dated.	R20Communicating change to schoolsFrom the field · not in the guide
Classroom floor	The minimum standard of learning, safety and consent that any test reaching students must meet. A term used in this guide.	R10Lean product developmentThe classroom floor · slide 44 (opens in a new tab)	British Educational Research Association. (2024). Ethical guidelines for educational research (5th ed.). bera.ac.uk
Classroom orchestration	A teacher's real-time management of several activities, groups and tools in a lesson. Teacher dashboards are often designed to support it.	R15Learning analyticsFrom the field · not in the guide	Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. doi
Clickable prototype	Linked screens that respond to taps or clicks but have no working logic behind them.	R07Design methods IIMatch the fidelity · slide 26 (opens in a new tab)	Houde, S., & Hill, C. (1997). What do prototypes prototype? In M. Helander, T. K. Landauer, & P. Prabhu (Eds.), Handbook of human-computer interaction (2nd ed., pp. 367–381). Elsevier. doi Rudd, J., Stern, K., & Isensee, S. (1996). Low vs. high-fidelity prototyping debate. Interactions, 3(1), 76–85. doi
Client–server model	The arrangement behind most web software: a client, such as a browser or app, sends requests, and a server holding the shared data and rules sends back responses.	R11Software development for non-engineersFrom the field · not in the guide	MDN Web Docs. (n.d.). How the web works. Mozilla. Retrieved September 17, 2026, from developer.mozilla.org Fielding, R. T., & Taylor, R. N. (2002). Principled design of the modern Web architecture. ACM Transactions on Internet Technology, 2(2), 115–150. doi
Closed question	A survey question answered by choosing from a fixed set of options. It is easy to count, but it can only return the answers its writer thought of.	R05Evaluative researchFrom the field · not in the guide	Pew Research Center. (2021). Writing survey questions. pewresearch.org Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press. archive.org
Closing the loop	Making sure analytics leads to an action that reaches learners (Clow, 2012). R22 Telling the person who reported a problem what happened to it, including when the answer is "not yet".	R15Learning analyticsThe action first · slide 26 (opens in a new tab) R22Support and feedback loopsClose the loop · slide 32 (opens in a new tab)	Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 134–138). ACM. doi Arizona State University. (2011). New customer-rage study: Fewer consumers satisfied than ever before [News release]. ASU News. news.asu.edu Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019). A large-scale implementation of predictive learning analytics in higher education: The teachers’ role and perspective. Educational Technology Research and Development, 67, 1273–1306. doi
Cloud computing	Rented computing resources that can be added or released quickly with little effort (Mell & Grance, 2011).	R11Software development for non-engineersThe spine · slide 15 (opens in a new tab)	Mell, P., & Grance, T. (2011). The NIST definition of cloud computing (NIST Special Publication 800-145). National Institute of Standards and Technology. doi
Cluster	A group whose members share an experience and tend to resemble each other: a class, a teacher's classes or a school.	R13Experimentation and product analyticsSize for clusters · slide 37 (opens in a new tab)	Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research. Arnold. wiley.com Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. doi Education Endowment Foundation. (2022). Statistical analysis guidance for EEF evaluations. d2tic4wvo1iusb.cloudfront.net
Cluster randomised trial	A trial that randomises whole groups, such as classes or schools, rather than individual pupils. It needs a larger sample than an individually randomised trial of the same precision.	R13Experimentation and product analyticsFrom the field · not in the guide	Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research. Arnold. wiley.com Education Endowment Foundation. (2022). Statistical analysis guidance for EEF evaluations. d2tic4wvo1iusb.cloudfront.net
Co-design	Designers and people not trained in design working together in the design process (Sanders & Stappers, 2008).	R03Design thinking, used honestlyProtect their time · slide 43 (opens in a new tab)	Sanders, E. B.-N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. CoDesign, 4(1), 5–18. doi Penuel, W. R., Roschelle, J., & Shechtman, N. (2007). Designing formative assessment software with teachers: An analysis of the co-design process. Research and Practice in Technology Enhanced Learning, 2(1), 51–74. doi Druin, A. (2002). The role of children in the design of new technology. Behaviour & Information Technology, 21(1), 1–25. doi
Co-evolution of problem and solution	The observation that designers do not fix the problem first and then solve it: their understanding of the problem and their ideas for a solution develop together, each reshaping the other.	R03Design thinking, used honestlyFrom the field · not in the guide	Dorst, K. (2011). The core of 'design thinking' and its application. Design Studies, 32(6), 521–532. doi Dorst, K., & Cross, N. (2001). Creativity in the design process: Co-evolution of problem–solution. Design Studies, 22(5), 425–437. doi
Coaching	Repeated cycles of observation, feedback and planning with an individual teacher or a small group.	R21Professional learning that changes practiceCoaching · slide 32 (opens in a new tab)	Kraft, M. A., Blazar, D., & Hogan, D. (2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research, 88(4), 547–588. doi Showers, B., & Joyce, B. (1996). The evolution of peer coaching. Educational Leadership, 53(6), 12–16. ascd.org Joyce, B., & Showers, B. (2002). Student achievement through staff development (3rd ed.). Association for Supervision and Curriculum Development. books.google.com
Code	A short label attached to a passage of data, used to find and compare similar passages.	R06Design methods ISynthesise together · slide 20 (opens in a new tab)	Saldaña, J. (2025). The coding manual for qualitative researchers (5th ed.). SAGE. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. doi
Code review	The reading of one developer's changes by another before they are merged, to find defects, improve the code and spread knowledge of it across the team.	R11Software development for non-engineersFrom the field · not in the guide	Bacchelli, A., & Bird, C. (2013). Expectations, outcomes, and challenges of modern code review. In Proceedings of the 35th International Conference on Software Engineering (pp. 712–721). IEEE. doi
Codebook	The list of codes used in an analysis, each with a definition, rules for when it applies and an example, kept so that coding stays consistent across people and time.	R06Design methods IFrom the field · not in the guide	Saldaña, J. (2025). The coding manual for qualitative researchers (5th ed.). SAGE.
Coded prototype	Working software built to answer a question, not to production standards.	R07Design methods IIMatch the fidelity · slide 26 (opens in a new tab)	Government Digital Service. (2016). Making prototypes. GOV.UK Service Manual. gov.uk Rudd, J., Stern, K., & Isensee, S. (1996). Low vs. high-fidelity prototyping debate. Interactions, 3(1), 76–85. doi Walker, M., Takayama, L., & Landay, J. A. (2002). High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 46(5), 661–665. doi
Coding reliability	An approach to thematic analysis that uses a fixed codebook and measures agreement between coders (Braun & Clarke, 2021).	R06Design methods ISynthesise together · slide 20 (opens in a new tab)	Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. doi Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. SAGE.
Cognitive interview	A survey pretesting method in which a few respondents answer draft questions while explaining how they understood them and how they arrived at each answer.	R05Evaluative researchFrom the field · not in the guide	Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. SAGE. doi
Cognitive load	The demand a task places on working memory (Sweller, 1988).	R08UX foundations and design systemsThe spine · slide 13 (opens in a new tab)	Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. doi Sweller, J., van Merriënboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–296. doi Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31(2), 261–292. doi
Cognitive offloading	Using an action or tool to reduce the thinking a task demands (Risko & Gilbert, 2016).	R16AI in learning productsThe spine · slide 13 (opens in a new tab)	Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688. doi Cash, T. N., Kelly, M. O., Macnamara, B. N., & Risko, E. F. (2026). Is AI making us stupid? Trends in Cognitive Sciences, 30(8), 673–675. doi Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1), Article 6. doi
Cognitive walkthrough	An inspection method in which reviewers step through a task as a first-time user would, asking at each step whether that user would know what to do and see that it had worked.	R05Evaluative researchFrom the field · not in the guide	Wharton, C., Rieman, J., Lewis, C., & Polson, P. (1994). The cognitive walkthrough method: A practitioner's guide. In J. Nielsen & R. L. Mack (Eds.), Usability inspection methods (pp. 105–140). Wiley.
Cohen's d	A standardised effect size that divides the difference between groups by the spread, so studies that use different tests can be compared (Cohen, 1988).	R18Basic statistics for educational technologistsEffect size · slide 30 (opens in a new tab)	Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. doi
Cohen's kappa	A measure of how far two raters agree when sorting items into categories, corrected for the agreement expected by chance.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. doi Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi
Coherence	Consistency between professional learning and the teacher's curriculum, knowledge, students and wider policy (Desimone, 2009).	R21Professional learning that changes practiceThe teacher's own lesson · slide 20 (opens in a new tab)	Desimone, L. M. (2009). Improving impact studies of teachers' professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199. doi Desimone, L. M., & Garet, M. S. (2015). Best practices in teachers' professional development in the United States. Psychology, Society, & Education, 7(3), 252–263. repositorio.ual.es Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915–945. doi
Coherence principle	People learn better when extraneous material is excluded rather than included (Mayer, 2024).	R08UX foundations and design systemsThe misconception · slide 9 (opens in a new tab)	Mayer, R. E. (2024). The past, present, and future of the cognitive theory of multimedia learning. Educational Psychology Review, 36(1), Article 8. doi Mayer, R. E. (2021). Multimedia learning (3rd ed.). Cambridge University Press. doi Sundararajan, N., & Adesope, O. (2020). Keep it coherent: A meta-analysis of the seductive details effect. Educational Psychology Review, 32(3), 707–734. doi
Cohort analysis	Analysis that groups users by a shared starting point, such as the week they joined, and follows each group over time, so that changes in behaviour are not hidden by a changing user mix.	R13Experimentation and product analyticsFrom the field · not in the guide	Croll, A., & Yoskovitz, B. (2013). Lean analytics: Use data to build a better startup faster. O'Reilly Media.
Colour contrast ratio	A measure from 1:1 to 21:1 of the difference in brightness between text and its background. WCAG level AA asks for at least 4.5:1 for normal-sized text.	R08UX foundations and design systemsFrom the field · not in the guide	World Wide Web Consortium. (2023). Web Content Accessibility Guidelines (WCAG) 2.2 (W3C Recommendation). w3.org
Commercial off-the-shelf (COTS) software	Ready-made software bought or licensed from a supplier and configured for use, as opposed to software built to order.	R01Digital product managementFrom the field · not in the guide	Hopson, M., McFadden, V., Refoy, R., & Rouault, A. (Eds.). (2020). De-risking government technology: Federal agency field guide. 18F, U.S. General Services Administration. guides.18f.gov
Communications plan	A schedule of who needs to be told what, by whom, through which channel and when, agreed before a change is announced.	R20Communicating change to schoolsFrom the field · not in the guide	Project Management Institute. (2017). A guide to the project management body of knowledge (PMBOK guide) (6th ed.). Project Management Institute.
Community of practice	A group of people who share a craft or concern and get better at it by interacting regularly.	R21Professional learning that changes practiceFrom the field · not in the guide	Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge University Press. doi Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press. doi
Compatibility	The degree to which an innovation is viewed as consistent with the current values, previous experiences and needs of prospective adopters. Late adopters weigh it heavily (Rogers, 2003).	R19Adoption and change in schoolsThe reluctant adopter · slide 27 (opens in a new tab)	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Complaint iceberg	The pattern in which about half of people with a problem tell a front-line person and only 1–5% escalate to head office, so head office hears from the tip (Goodman, 1999).	R22Support and feedback loopsThe misconception · slide 10 (opens in a new tab)	Goodman, J. (1999). Basic facts on customer complaint behavior and the impact of service on the bottom line. Competitive Advantage, June, 1–5. newtoncomputing.com
Complier average causal effect (CACE)	An estimate of the effect of an intervention on those who took it up as intended, reported alongside the intention-to-treat estimate.	R13Experimentation and product analyticsFrom the field · not in the guide	Education Endowment Foundation. (2022). Statistical analysis guidance for EEF evaluations. d2tic4wvo1iusb.cloudfront.net Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455. doi
Component	A reusable interface part, such as a button or text field, with agreed behaviour and code.	R08UX foundations and design systemsSystem first · slide 31 (opens in a new tab)	Frost, B. (2016). Atomic design. Brad Frost Web. atomicdesign.bradfrost.com Government Digital Service. (n.d.-a). Contribution criteria. GOV.UK Design System. Retrieved September 17, 2026, from design-system.service.gov.uk
Computer-assisted qualitative data analysis software (CAQDAS)	Software for storing, coding, searching and retrieving qualitative data. It organises the analyst's work and does not do the analysis.	R06Design methods IFrom the field · not in the guide	Saldaña, J. (2025). The coding manual for qualitative researchers (5th ed.). SAGE.
Concerns-Based Adoption Model	A model developed at the University of Texas at Austin that describes the stages of concern and levels of use a person moves through as they meet a change (Hall & Hord, 2014).	R19Adoption and change in schoolsThe reluctant adopter · slide 28 (opens in a new tab)	Hall, G. E., & Hord, S. M. (2014). Implementing change: Patterns, principles, and potholes (4th ed.). Pearson. pearson.com American Institutes for Research. (n.d.). Stages of concern: Concerns-Based Adoption Model. Retrieved September 17, 2026, from air.org
Concierge test	A test in which people openly deliver by hand what a product would later automate.	R10Lean product developmentThe smallest test · slide 26 (opens in a new tab)	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com Bland, D. J., & Osterwalder, A. (2019). Testing business ideas: A field guide for rapid experimentation. Wiley. books.google.com
Cone of uncertainty	The range within which a skilled estimate can be wrong, widest at the start of a project (McConnell, 2006).	R09Agile and Scrum in practiceMake work visible · slide 28 (opens in a new tab)	McConnell, S. (2006). Software estimation: Demystifying the black art. Microsoft Press. Boehm, B. W. (1981). Software engineering economics. Prentice-Hall.
Confabulation	Confident but false content produced by a generative model, often called hallucination (National Institute of Standards and Technology, 2024).	R16AI in learning productsGround and check · slide 47 (opens in a new tab)	National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. doi Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), Article 248. doi
Confidence interval	A range of plausible values for a true figure, built by a method that captures it a stated share of the time.	R18Basic statistics for educational technologistsUncertainty · slide 23 (opens in a new tab)	Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. doi Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. doi
Confidentiality, integrity and availability (CIA triad)	The three aims of information security: data is seen only by those allowed, is not altered without authority, and is there when needed.	R17Privacy, security and ethicsFrom the field · not in the guide	National Institute of Standards and Technology. (2004). Standards for security categorization of federal information and information systems (FIPS PUB 199). National Institute of Standards and Technology. Anderson, R. (2020). Security engineering: A guide to building dependable distributed systems (3rd ed.). Wiley. doi
Configuration drift	The gradual divergence of servers or environments that are meant to be identical, caused by manual, unrecorded changes. It makes releases behave differently from one environment to the next.	R12DevOps and continuous deliveryFrom the field · not in the guide	Morris, K. (2016). Infrastructure as code: Managing servers in the cloud. O'Reilly Media.
Configuration management	Keeping everything needed to build and run a system, including code, settings, scripts and environment definitions, identified, versioned and reproducible, so that any release can be recreated exactly.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Confirmation bias	Seeking or reading evidence in ways that favour a belief already held (Nickerson, 1998).	R04Discovery researchNotes and meanings · slide 38 (opens in a new tab)	Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. doi Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology, 12(3), 129–140. doi
Confounder	A factor that influences both a supposed cause and its outcome, creating a misleading association.	R18Basic statistics for educational technologistsRelationship and cause · slide 38 (opens in a new tab)	Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books. Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press.
Consent	Informed, voluntary agreement to take part, which can be withdrawn at any time.	R04Discovery researchTwo yeses · slide 43 (opens in a new tab)	British Educational Research Association. (2024). Ethical guidelines for educational research (5th ed.). bera.ac.uk Personal Data Protection Commission. (2024). Advisory guidelines on the PDPA for children's personal data in the digital environment. pdpc.gov.sg
Contact rate	The number of support contacts divided by the number of active users or accounts in the same period, used to tell growth in use from growth in problems.	R22Support and feedback loopsFrom the field · not in the guide
Container	The screens, navigation and controls that hold a learning product's content.	R07Design methods IIDesign the content · slide 38 (opens in a new tab)	Richards, S. (2017). Content design. Content Design London. contentdesign.london
Container orchestration	Software that starts, places, scales, restarts and updates containers across a group of servers automatically, to match a declared desired state. Kubernetes is the most widely used.	R12DevOps and continuous deliveryFrom the field · not in the guide	Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. Communications of the ACM, 59(5), 50–57. doi
Contamination	Spillover of one version's effects into the other group, for example through shared answers or a shared lesson.	R13Experimentation and product analyticsThe right unit · slide 25 (opens in a new tab)	Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research. Arnold. wiley.com Ritter, S., Murphy, A., Fancsali, S. E., Fitkariwala, V., Patel, N., & Lomas, J. D. (2020). UpGrade: An open source tool to support A/B testing in educational software [Paper presentation]. Workshop on Educational A/B Testing at Scale, Learning @ Scale 2020. upgradeplatform.org
Content analysis	A systematic method that assigns passages of text or other material to defined categories and often counts them, so that inferences can be checked and repeated by others.	R06Design methods IFrom the field · not in the guide	Krippendorff, K. (2018). Content analysis: An introduction to its methodology (4th ed.). SAGE. doi
Content design	Meeting a user need with the right content, in the right format and place (Winters, 2019). R20 The practice of helping people quickly find out what they need to know or do, a test of the reader's position after the message, not of the sender's activity before it (Government Digital Service, n.d.-e).	R07Design methods IIDesign the content · slide 40 (opens in a new tab) R20Communicating change to schoolsThe misconception · slide 8 (opens in a new tab)	Richards, S. (2017). Content design. Content Design London. contentdesign.london Government Digital Service. (n.d.-e). Understand content design. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk Winters, S. (2019). What is content design? Content Design London. contentdesign.london
Content prototype	A real lesson, task or worked example, tried with learners, to test the content itself.	R07Design methods IIDesign the content · slide 39 (opens in a new tab)	Tripp, S. D., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative instructional design strategy. Educational Technology Research and Development, 38(1), 31–44. doi Richards, S. (2017). Content design. Content Design London. contentdesign.london
Context of use	The users, goals, tasks, equipment and environment in which a product is used (International Organization for Standardization, 2018).	R05Evaluative researchThe spine · slide 15 (opens in a new tab)	International Organization for Standardization. (2018). Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts (ISO Standard No. 9241-11:2018). iso.org
Context window	The most text, counted in tokens, that a model can take into account at once, including the prompt, any supplied documents and its own reply.	R16AI in learning productsFrom the field · not in the guide
Context-specific repetition	Practising a new behaviour repeatedly in the setting where it will be used, so it becomes a habit.	R21Professional learning that changes practiceThe second week · slide 47 (opens in a new tab)	Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov
Contextual inquiry	Observation and interviewing of people doing their work, where they usually do it (Beyer & Holtzblatt, 1998).	R04Discovery researchWatch first · slide 20 (opens in a new tab)	Beyer, H., & Holtzblatt, K. (1998). Contextual design: Defining customer-centered systems. Morgan Kaufmann. archive.org Holtzblatt, K., & Beyer, H. R. (2013). Contextual design. In M. Soegaard & R. F. Dam (Eds.), The encyclopedia of human-computer interaction (2nd ed.). Interaction Design Foundation. ixdf.org Flaherty, K. (2020). Contextual inquiry: Inspire design by observing and interviewing users in their context. Nielsen Norman Group. nngroup.com
Contextual integrity	The view that privacy is kept when information flows according to the norms of the setting it was shared in, and breached when it moves to a setting with other norms.	R17Privacy, security and ethicsFrom the field · not in the guide	Nissenbaum, H. (2010). Privacy in context: Technology, policy, and the integrity of social life. Stanford University Press. doi
Continuous delivery	Building software so that it can be released to production at any time (Fowler, 2013).	R12DevOps and continuous deliveryAutomate the path · slide 20 (opens in a new tab)	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com Fowler, M. (2013). Continuous delivery. martinfowler.com. martinfowler.com Humble, J. (n.d.). What is continuous delivery? Continuous Delivery. Retrieved September 17, 2026, from continuousdelivery.com
Continuous deployment	Sending every change that passes the pipeline to production automatically, with no human release decision.	R12DevOps and continuous deliveryAutomate the path · slide 20 (opens in a new tab)	Fowler, M. (2013). Continuous delivery. martinfowler.com. martinfowler.com Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Continuous integration	Merging every developer's work into the main line at least daily, with each merge built and tested automatically (Fowler, 2024).	R12DevOps and continuous deliveryAutomate the path · slide 20 (opens in a new tab)	Fowler, M. (2024). Continuous integration. martinfowler.com. (Original work published 2000) martinfowler.com Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Contribution model	The published route by which teams propose, build and add new parts to a design system.	R08UX foundations and design systemsRun it as a product · slide 44 (opens in a new tab)	Government Digital Service. (n.d.-a). Contribution criteria. GOV.UK Design System. Retrieved September 17, 2026, from design-system.service.gov.uk Government Digital Service. (n.d.-b). Community. GOV.UK Design System. Retrieved September 17, 2026, from design-system.service.gov.uk Noakes, A., & Paul, T. (2017). Building the GOV.UK Design System. Government Digital Service blog. gds.blog.gov.uk
Control group	A group like the one that got the change, in the same weeks, that did not get it. It shows how much of a rise would have happened anyway.	R13Experimentation and product analyticsThe spine · slide 13 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0 (WWC 2022008). U.S. Department of Education, Institute of Education Sciences. ies.ed.gov
Convergent thinking	Narrowing many options down to a focused choice.	R03Design thinking, used honestlyThe spine · slide 14 (opens in a new tab)	Design Council. (n.d.-a). Framework for innovation. Retrieved September 17, 2026, from designcouncil.org.uk Design Council. (2007). Eleven lessons: Managing design in eleven global brands. A study of the design process. idi-design.ie
Conway's law	The observation that a system's structure comes to mirror the communication structure of the organisation that built it: three contracted teams tend to produce a system in three parts.	R11Software development for non-engineersFrom the field · not in the guide	Brooks, F. P., Jr. (1975). The mythical man-month: Essays on software engineering. Addison-Wesley. Anniversary edition published 1995. informit.com Conway, M. E. (1968). How do committees invent? Datamation, 14(4), 28–31. Skelton, M., & Pais, M. (2019). Team topologies: Organizing business and technology teams for fast flow. IT Revolution Press.
Core components	The parts of an approach that must stay the same for it to work; also called active ingredients (Sharples et al., 2024).	R19Adoption and change in schoolsExpect variation · slide 42 (opens in a new tab)	Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (FMHI Publication No. 231). University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.
Core features of effective professional development	Five features set out by Desimone: content focus, active learning, coherence, sustained duration and collective participation (Desimone, 2009).	R21Professional learning that changes practiceThe teacher's own lesson · slide 20 (opens in a new tab)	Desimone, L. M. (2009). Improving impact studies of teachers' professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199. doi Desimone, L. M., & Garet, M. S. (2015). Best practices in teachers' professional development in the United States. Psychology, Society, & Education, 7(3), 252–263. repositorio.ual.es Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915–945. doi
Correlation coefficient	A number from −1 to +1 that shows how closely two variables move together in a straight line. Usually written r, it says nothing about what causes what.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Rodgers, J. L., & Nicewander, W. A. (1988). Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1), 59–66. doi
Cost of delay	What it costs, per unit of time, to have a piece of work not yet done. R10 What it costs to have a result or a feature later rather than sooner (Reinertsen, 2009).	R01Digital product managementSay no in public · slide 31 (opens in a new tab) R10Lean product developmentWork in progress · slide 33 (opens in a new tab)	Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com Arnold, J. (n.d.). Cost of delay divided by duration. Black Swan Farming. blackswanfarming.com Scaled Agile, Inc. (n.d.). WSJF. SAFe Framework. framework.scaledagile.com
Counterfactual	What would have happened to the same people without the change. Never observed; estimated by a control group. R18 What would have happened without the intervention. Never observed directly.	R13Experimentation and product analyticsThe spine · slide 13 (opens in a new tab) R18Basic statistics for educational technologistsRelationship and cause · slide 36 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. doi
Course Signals	Purdue’s early-warning system, which gave students a red, yellow or green signal based on performance, effort and background. Its widely cited retention gain was a correlation, not an established effect (Arnold and Pistilli, 2012).	R15Learning analyticsThe action first · slide 27 (opens in a new tab)	Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267–270). ACM. doi Caulfield, M. (2013). Why the Course Signals math does not add up. Hapgood. hapgood.us Clow, D. (2013). Looking harder at Course Signals. Doug Clow’s Imaginatively-Titled Blog. dougclow.org
Crazy Eights	A sketching exercise in which each person folds a sheet into eight panels and draws eight variations of an idea in eight minutes.	R07Design methods IIFrom the field · not in the guide	Knapp, J., Zeratsky, J., & Kowitz, B. (2016). Sprint: How to solve big problems and test new ideas in just five days. Simon & Schuster.
Creative confidence	The belief that one can come up with new ideas and the willingness to act on them, treated as something that can be built through practice.	R03Design thinking, used honestlyFrom the field · not in the guide	Kelley, T., & Kelley, D. (2013). Creative confidence: Unleashing the creative potential within us all. Crown Business.
Critical incident	A specific, observable occasion that a participant can describe in detail (Flanagan, 1954).	R04Discovery researchThe last time · slide 25 (opens in a new tab)	Flanagan, J. C. (1954). The critical incident technique. Psychological Bulletin, 51(4), 327–358. doi Butterfield, L. D., Borgen, W. A., Amundson, N. E., & Maglio, A.-S. T. (2005). Fifty years of the critical incident technique: 1954–2004 and beyond. Qualitative Research, 5(4), 475–497. doi
Critical Response Process	A four-step critique protocol that puts the maker's questions before responders' opinions (Lerman, n.d.).	R07Design methods IICritique against the goal · slide 33 (opens in a new tab)	Lerman, L. (n.d.). Critical Response Process. Liz Lerman. Retrieved September 18, 2026, from lizlerman.com Lerman, L., & Borstel, J. (2003). Liz Lerman's Critical Response Process: A method for getting useful feedback on anything you make, from dance to dessert. Liz Lerman Dance Exchange.
Critique	Analysis of whether a design is expected to achieve its goals (Connor, 2016).	R07Design methods IIThe misconception · slide 11 (opens in a new tab)	Connor, A., & Irizarry, A. (2015). Discussing design: Improving communication and collaboration through critique. O'Reilly Media. oreilly.com Connor, A. (2016). 3 kinds of feedback. Discussing Design, Medium. medium.com
Cronbach's alpha	A figure from 0 to 1 showing how consistently the items in a test or questionnaire measure the same thing. Values of about 0.7 and above are usually accepted.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. doi Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
Cross-border data transfer	Sending personal data to, or allowing access to it from, another country. Many laws permit this only if the data stays comparably protected, which affects where a vendor may host.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Personal Data Protection Act 2012 (2020 Rev. Ed.) (Singapore).
Cross-functional team	A team whose members together hold all the skills needed to turn a backlog item into working product, without handing work to another team.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Cultural probe	A kit of open, playful tasks, such as postcards, maps and a camera, given to people to complete on their own and return, to inspire designers with glimpses of their lives.	R03Design thinking, used honestlyFrom the field · not in the guide	Gaver, B., Dunne, T., & Pacenti, E. (1999). Design: Cultural probes. Interactions, 6(1), 21–29. doi
Cumulative flow diagram	A stacked chart of how many work items sit in each stage over time, used to spot bottlenecks and growing work in progress.	R09Agile and Scrum in practiceFrom the field · not in the guide	Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Blue Hole Press. books.google.com Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com
CUPED	A variance reduction method that adjusts each user's outcome using data from before the experiment, so that smaller effects can be detected with the same sample.	R13Experimentation and product analyticsFrom the field · not in the guide	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi Deng, A., Xu, Y., Kohavi, R., & Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining (pp. 123–132). ACM. doi
Current-state and future-state maps	A current-state map shows an experience as research found it. A future-state map shows the experience the team intends to create.	R06Design methods IFrom the field · not in the guide	Gibbons, S. (2018). Journey mapping 101. Nielsen Norman Group. nngroup.com Kalbach, J. (2020). Mapping experiences: A complete guide to customer alignment through journeys, blueprints, and diagrams (2nd ed.). O'Reilly Media. openlibrary.org
Curse of knowledge	The difficulty people have in setting aside what they know when judging what others know (Camerer et al., 1989).	R20Communicating change to schoolsThe spine · slide 15 (opens in a new tab)	Camerer, C., Loewenstein, G., & Weber, M. (1989). The curse of knowledge in economic settings: An experimental analysis. Journal of Political Economy, 97(5), 1232–1254. doi Newton, E. (1990). Overconfidence in the communication of intent: Heard and unheard melodies [Unpublished doctoral dissertation]. Stanford University. Heath, C., & Heath, D. (2007). Made to stick: Why some ideas survive and others die. Random House. heathbrothers.com
Custodian	The person or team that runs the systems a dataset is stored in, including security, backup and deletion.	R14Data strategy and governanceNamed owners · slide 25 (opens in a new tab)	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Privacy Technical Assistance Center. (2015). Data governance and stewardship (Rev. ed.). U.S. Department of Education. studentprivacy.ed.gov
Customer development	Testing a business idea's hypotheses with potential users and buyers before building at scale (Blank, 2013).	R10Lean product developmentThe misconception · slide 10 (opens in a new tab)	Blank, S. (2013). Why the lean start-up changes everything. Harvard Business Review, 91(5), 63–72. hbr.org Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com
Customer effort	How hard a person has to work to get a problem solved, including repeat contacts and channel switches (Dixon et al., 2010).	R22Support and feedback loopsHelp in place · slide 27 (opens in a new tab)	Dixon, M., Freeman, K., & Toman, N. (2010). Stop trying to delight your customers. Harvard Business Review, 88(7/8). hbr.org Dixon, M., Toman, N., & DeLisi, R. (2013). The effortless experience: Conquering the new battleground for customer loyalty. Portfolio/Penguin.
Customer satisfaction score (CSAT)	The share of users who rate a support interaction as satisfactory, usually from a one-question survey sent when the ticket closes.	R22Support and feedback loopsFrom the field · not in the guide
Cycle time	The time a work item takes from the moment work starts on it to the moment it is finished.	R09Agile and Scrum in practiceFrom the field · not in the guide	Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Blue Hole Press. books.google.com Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com
Cynefin framework	A distinction between simple contexts, where leaders sense, categorise and respond, complicated ones, where they analyse with expert help, and complex ones, where they probe, then sense and respond (Snowden & Boone, 2007).	R03Design thinking, used honestlyStay in the problem · slide 21 (opens in a new tab)	Snowden, D. J., & Boone, M. E. (2007). A leader's framework for decision making. Harvard Business Review, 85(11), 68–76. hbr.org
D
Daily Scrum	A 15-minute daily meeting at which a Scrum team's developers inspect progress towards the Sprint Goal and plan the next day's work. Many teams call it the stand-up.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Dark data	Stored data whose value is unknown. In a 2016 survey, organisations described 52% of what they stored this way, and another 33% as redundant, obsolete or trivial (Veritas Technologies, 2016).	R14Data strategy and governanceThe misconception · slide 9 (opens in a new tab)	Veritas Technologies. (2016). Veritas Global Databerg Report finds 85% of stored data is either dark, or redundant, obsolete, or trivial (ROT) [Press release]. veritas.com
Dark launch	Running new code in production for real users without them being able to tell (Fowler, 2020).	R12DevOps and continuous deliveryDeploy is not release · slide 31 (opens in a new tab)	Fowler, M. (2020). Dark launching. martinfowler.com. martinfowler.com
Dark pattern	An interface design that steers people into choices they would not otherwise make, such as a pre-ticked consent box or a hidden opt-out.	R17Privacy, security and ethicsFrom the field · not in the guide	Gray, C. M., Kou, Y., Battles, B., Hoggatt, J., & Toombs, A. L. (2018). The dark (patterns) side of UX design. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Paper 534). ACM. doi Mathur, A., Acar, G., Friedman, M. J., Lucherini, E., Mayer, J., Chetty, M., & Narayanan, A. (2019). Dark patterns at scale: Findings from a crawl of 11K shopping websites. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), Article 81. doi
Dashboard	A screen that summarises data for a teacher, a leader or a student.	R15Learning analyticsThe action first · slide 25 (opens in a new tab)	Jivet, I., Scheffel, M., Specht, M., & Drachsler, H. (2018). License to evaluate: Preparing learning analytics dashboards for educational practice. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 31–40). ACM. doi Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 134–138). ACM. doi
Data catalogue	A searchable inventory of an organisation's datasets, recording what each holds, where it is stored, who owns it and how to request access.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Data cleansing	Correcting or removing data that is wrong, incomplete, duplicated or badly formatted. It treats the symptom; fixing the point of entry treats the cause.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Data controller and data processor	The controller decides why and how personal data is processed; the processor handles it on the controller's behalf. A school is usually the controller and its edtech vendor the processor.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Personal Data Protection Act 2012 (2020 Rev. Ed.) (Singapore).
Data dictionary	A document listing each field in a dataset with its name, meaning, format, allowed values and source, so that everyone reads the data the same way.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Data governance	Policies and procedures covering data across its life, "from acquisition to use to disposal" (Privacy Technical Assistance Center, 2015).	R14Data strategy and governanceNamed owners · slide 25 (opens in a new tab)	Privacy Technical Assistance Center. (2015). Data governance and stewardship (Rev. ed.). U.S. Department of Education. studentprivacy.ed.gov DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org OECD. (2019). The path to becoming a data-driven public sector. OECD Publishing. doi
Data governance council	The cross-department group that sets data policy, resolves disputes between owners and approves standards. Also called a data governance committee or board.	R14Data strategy and governanceFrom the field · not in the guide	Privacy Technical Assistance Center. (2015). Data governance and stewardship (Rev. ed.). U.S. Department of Education. studentprivacy.ed.gov DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Data lake	A store that holds large volumes of data in its raw, original form, structured or not, leaving it to be organised when it is used.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Dehghani, Z. (2019). How to move beyond a monolithic data lake to a distributed data mesh. martinfowler.com. martinfowler.com
Data lifecycle	The stages a dataset passes through, from planning and collection, through storage, use and sharing, to archiving or deletion.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Privacy Technical Assistance Center. (2015). Data governance and stewardship (Rev. ed.). U.S. Department of Education. studentprivacy.ed.gov
Data literacy	The ability to read, question, interpret and act on data. For educators, it means turning data into teaching decisions.	R14Data strategy and governanceFrom the field · not in the guide	Mandinach, E. B., & Gummer, E. S. (2016). Data literacy for educators: Making it count in teacher preparation and practice. Teachers College Press.
Data maturity assessment	A structured review that rates an organisation's data management practices against defined levels, to show where it stands and what to improve first.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Data mesh	A model in which domain teams own their data and serve it to others as a product, on a shared platform and under federated rules (Dehghani, 2022).	R14Data strategy and governanceNamed owners · slide 26 (opens in a new tab)	Dehghani, Z. (2019). How to move beyond a monolithic data lake to a distributed data mesh. martinfowler.com. martinfowler.com Dehghani, Z. (2022). Data mesh: Delivering data-driven value at scale. O'Reilly Media.
Data minimisation	Collecting only the personal data that is adequate, relevant and necessary for a stated purpose (European Union, 2016).	R14Data strategy and governanceCollect less · slide 34 (opens in a new tab)	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. eur-lex.europa.eu Information Commissioner's Office. (2020). Age appropriate design: A code of practice for online services (Standard 8: Data minimisation). ico.org.uk
Data model	A description of the things a system holds data about, their attributes and how they relate, such as pupils, classes and enrolments.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Kleppmann, M. (2017). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. O'Reilly Media.
Data pipeline	An automated series of steps that moves data from where it is produced to where it is used, transforming and checking it on the way.	R14Data strategy and governanceFrom the field · not in the guide	Kleppmann, M. (2017). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. O'Reilly Media.
Data processing agreement	The contract that binds a vendor to handle personal data only on the school's or ministry's instructions, keep it secure, name its sub-processors and return or delete it at the end.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88.
Data product	A dataset prepared and maintained for others to use, with documentation, quality guarantees and a named owner, as a team would treat any product.	R14Data strategy and governanceFrom the field · not in the guide	Dehghani, Z. (2022). Data mesh: Delivering data-driven value at scale. O'Reilly Media.
Data profiling	Examining a dataset's actual contents, such as value ranges, blanks, formats and duplicates, to find quality problems before the data is used.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Data protection impact assessment (DPIA)	A documented assessment, made before a project starts, of the risks that its use of personal data poses to people, and of the measures that will reduce them.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Article 29 Data Protection Working Party. (2017). Guidelines on data protection impact assessment (DPIA) and determining whether processing is "likely to result in a high risk" for the purposes of Regulation 2016/679 (WP 248 rev.01). European Commission.
Data protection officer (DPO)	The person an organisation appoints to oversee its compliance with data protection law, advise on impact assessments and deal with the regulator and the public.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Personal Data Protection Act 2012 (2020 Rev. Ed.) (Singapore).
Data quality	How fit data is for a stated purpose, judged on dimensions such as completeness and accuracy (Government Data Quality Hub, 2020).	R14Data strategy and governanceCollect less · slide 31 (opens in a new tab)	Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33. doi Government Data Quality Hub. (2020). The Government Data Quality Framework. GOV.UK. gov.uk Nagle, T., Redman, T. C., & Sammon, D. (2017). Only 3% of companies' data meets basic quality standards. Harvard Business Review. hbr.org
Data quality dimensions	The aspects on which data quality is assessed, commonly accuracy, completeness, uniqueness, consistency, timeliness and validity.	R14Data strategy and governanceFrom the field · not in the guide	Government Data Quality Hub. (2020). The Government Data Quality Framework. GOV.UK. gov.uk Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33. doi DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Data subject rights	The rights people have over data about them, such as to see it, correct it, have it erased, object to its use and take it elsewhere.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Personal Data Protection Act 2012 (2020 Rev. Ed.) (Singapore).
Data warehouse	A central store of cleaned, structured data drawn from an organisation's operational systems and arranged for reporting and analysis, not for day-to-day transactions.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Kimball, R., & Ross, M. (2013). The data warehouse toolkit: The definitive guide to dimensional modeling (3rd ed.). Wiley.
Data-driven decision making	The systematic collection and analysis of data to guide decisions in schools and systems. Many now prefer "data-informed", to keep professional judgement in the picture.	R14Data strategy and governanceFrom the field · not in the guide	Mandinach, E. B. (2012). A perfect time for data use: Using data-driven decision making to inform practice. Educational Psychologist, 47(2), 71–85. doi
Database	The part of a system that stores records, such as students, classes and marks.	R11Software development for non-engineersThe spine · slide 15 (opens in a new tab)	Codd, E. F. (1970). A relational model of data for large shared data banks. Communications of the ACM, 13(6), 377–387. doi
De-implementation	Stopping an approach on purpose, with the same care as starting one (Sharples et al., 2024).	R19Adoption and change in schoolsPlan the exit · slide 47 (opens in a new tab)	Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk
Debrief	A short meeting of the research team straight after a session to record what each person noticed, before memory fades and sessions blur together.	R04Discovery researchFrom the field · not in the guide	Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com
Decision register	A table of the decisions a data collection serves, with each decision's owner, date, data needs and threshold.	R14Data strategy and governanceThe decision first · slide 18 (opens in a new tab)	Department for Education. (2018). Making data work: Report of the Teacher Workload Advisory Group. assets.publishing.service.gov.uk
Deep learning	Machine learning with neural networks of many layers, each layer building more abstract features from the one before. It underlies modern speech, image and language models.	R16AI in learning productsFrom the field · not in the guide	LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. doi
Definition of Done	The shared quality standard a piece of work must meet to count as part of the Increment.	R09Agile and Scrum in practiceThe spine · slide 16 (opens in a new tab)	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Definition of Ready	A team's agreed checklist for what a backlog item needs, such as clear acceptance criteria and a size estimate, before it can be taken into a sprint.	R09Agile and Scrum in practiceFrom the field · not in the guide	Rubin, K. S. (2012). Essential Scrum: A practical guide to the most popular agile process. Addison-Wesley.
Deliberate practice	Practice designed to improve one specific part of performance, with a clear goal, full concentration, immediate feedback and repetition.	R21Professional learning that changes practiceFrom the field · not in the guide	Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406. doi
Delivery	The work of building and releasing something reliably.	R01Digital product managementThe spine · slide 15 (opens in a new tab)	Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley. Sy, D. (2007). Adapting usability investigations for agile user-centered design. Journal of Usability Studies, 2(3), 112–132. uxpajournal.org Patton, J. (2017, May 10). Dual track development is not duel track. Jeff Patton & Associates. jpattonassociates.com
Deploy	To put a new version of software into production. It need not be visible to users.	R12DevOps and continuous deliveryDeploy is not release · slide 30 (opens in a new tab)	Government Digital Service. (2024). Deploying software regularly. GOV.UK Service Manual. gov.uk Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Deployment frequency	A count of deployments over a period; one of the three DORA measures of throughput (DORA, 2026).	R12DevOps and continuous deliveryThe dora measures · slide 37 (opens in a new tab)	DORA. (2026). DORA's software delivery performance metrics. Google Cloud. dora.dev Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com Harvey, N. (2026). A history of DORA's software delivery metrics. DORA. dora.dev
Deployment pipeline	The automated stages a change passes through from commit to production (Humble & Farley, 2010).	R12DevOps and continuous deliveryAutomate the path · slide 19 (opens in a new tab)	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com Government Digital Service. (2024). Deploying software regularly. GOV.UK Service Manual. gov.uk
Deployment rework rate	The share of deployments that are unplanned and made because of an incident in production (DORA, 2026).	R12DevOps and continuous deliveryThe dora measures · slide 37 (opens in a new tab)	DORA. (2026). DORA's software delivery performance metrics. Google Cloud. dora.dev Harvey, N. (2026). A history of DORA's software delivery metrics. DORA. dora.dev
Deprecation	A warning that a feature will be removed in a later release, given before it goes.	R20Communicating change to schoolsWhat was removed · slide 38 (opens in a new tab)	Lacan, O. (2026). Keep a Changelog (Version 2.0.0). keepachangelog.com Wu, J., He, H., Gao, K., Xiao, W., Li, J., & Zhou, M. (2024). A comprehensive analysis of challenges and strategies for software release notes on GitHub. Empirical Software Engineering, 29, Article 104. doi
Design brief	The written starting point of a design project, giving the problem as first understood, the people it is for, the constraints and what would count as success.	R03Design thinking, used honestlyFrom the field · not in the guide	Brown, T. (2009). Change by design: How design thinking transforms organizations and inspires innovation. HarperBusiness.
Design effect	The factor by which clustering inflates the sample needed: 1 + (m − 1) × ICC (Donner & Klar, 2000).	R13Experimentation and product analyticsSize for clusters · slide 38 (opens in a new tab)	Kish, L. (1965). Survey sampling. Wiley. wiley.com Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research. Arnold. wiley.com Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. doi
Design fixation	Blind adherence to a set of ideas that limits the options a designer considers (Jansson & Smith, 1991).	R07Design methods IISketch many · slide 21 (opens in a new tab)	Jansson, D. G., & Smith, G. M. (1991). Design fixation. Design Studies, 12(1), 3–11. doi Purcell, A. T., & Gero, J. S. (1996). Design and other types of fixation. Design Studies, 17(4), 363–383. doi
Design partner	A child or adult who takes part in design as an equal stakeholder, not only as a tester (Druin, 2002).	R03Design thinking, used honestlyProtect their time · slide 46 (opens in a new tab)	Druin, A. (2002). The role of children in the design of new technology. Behaviour & Information Technology, 21(1), 1–25. doi
Design review	A meeting held to approve a design or check it against requirements before it moves on. It is a gate, unlike a critique, which aims to improve work still in progress.	R07Design methods IIFrom the field · not in the guide	Connor, A., & Irizarry, A. (2015). Discussing design: Improving communication and collaboration through critique. O'Reilly Media. oreilly.com
Design space	The full set of possible designs for a problem. Sketches and prototypes are ways of exploring it before settling on one region.	R07Design methods IIFrom the field · not in the guide	Lim, Y.-K., Stolterman, E., & Tenenberg, J. (2008). The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Transactions on Computer-Human Interaction, 15(2), Article 7. doi Buxton, B. (2007). Sketching user experiences: Getting the design right and the right design. Morgan Kaufmann. books.google.com
Design sprint	A five-day process in which a small team maps a problem, sketches solutions, picks one, builds a realistic prototype and tests it with five users.	R07Design methods IIFrom the field · not in the guide	Knapp, J., Zeratsky, J., & Kowitz, B. (2016). Sprint: How to solve big problems and test new ideas in just five days. Simon & Schuster.
Design studio	A workshop in which a mixed team sketches ideas individually, presents and critiques them, then converges on a shared design over several rounds.	R07Design methods IIFrom the field · not in the guide	Gothelf, J., & Seiden, J. (2013). Lean UX: Applying lean principles to improve user experience. O'Reilly Media.
Design system	A product that holds an organisation's shared interface decisions, parts, guidance and code, with a team that maintains them.	R08UX foundations and design systemsThe misconception · slide 10 (opens in a new tab)	Frost, B. (2016). Atomic design. Brad Frost Web. atomicdesign.bradfrost.com Curtis, N. (2015). Team models for scaling a design system. EightShapes. medium.com Noakes, A., & Paul, T. (2017). Building the GOV.UK Design System. Government Digital Service blog. gds.blog.gov.uk
Design theatre	The visible rituals of design work, such as sticky notes and sprints, without the evidence and judgement behind them.	R03Design thinking, used honestlyThe misconception · slide 8 (opens in a new tab)	Kimbell, L. (2011). Rethinking design thinking: Part I. Design and Culture, 3(3), 285–306. doi Vinsel, L. (2017). Design thinking is kind of like syphilis: It's contagious and rots your brains. Medium. sts-news.medium.com Kolko, J. (2018). The divisiveness of design thinking. Interactions, 25(3), 28–34. doi
Design thinking	A way of framing and exploring a problem before solving it, drawn from how designers work.	R03Design thinking, used honestlyThe spine · slide 13 (opens in a new tab)	Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92. hbr.org Dorst, K. (2011). The core of 'design thinking' and its application. Design Studies, 32(6), 521–532. doi Kimbell, L. (2011). Rethinking design thinking: Part I. Design and Culture, 3(3), 285–306. doi
Design token	A named design decision, such as a colour or spacing value, stored once and used everywhere (Design Tokens Community Group, 2025).	R08UX foundations and design systemsSystem first · slide 31 (opens in a new tab)	Curtis, N. (2016). Tokens in design systems. EightShapes. medium.com Design Tokens Community Group. (2025). Design tokens format module 2025.10 (Final Community Group Report). W3C Community Group. designtokens.org
Designerly ways of knowing	The claim that design has its own forms of knowledge and problem solving, distinct from the sciences and the humanities, centred on proposing solutions to ill-defined problems.	R03Design thinking, used honestlyFrom the field · not in the guide	Cross, N. (1982). Designerly ways of knowing. Design Studies, 3(4), 221–227. doi
DesignOps	The orchestration and optimisation of people, processes and craft to increase the value and impact of design at scale (Kaplan, 2019).	R08UX foundations and design systemsRun it as a product · slide 45 (opens in a new tab)	Kaplan, K. (2019). DesignOps 101. Nielsen Norman Group. nngroup.com Kaplan, K. (2020). DesignOps maturity: Low in most organizations. Nielsen Norman Group. nngroup.com
Desirability, feasibility and viability	Three kinds of assumption in a plan: whether teachers and students want it, whether the team can build and run it, and whether anyone will pay and keep paying (Bland & Osterwalder, 2019).	R10Lean product developmentThe spine · slide 12 (opens in a new tab)	Bland, D. J., & Osterwalder, A. (2019). Testing business ideas: A field guide for rapid experimentation. Wiley. books.google.com Bland, D. J. (2020, August 4). How assumptions mapping can focus your teams on running experiments that matter. Strategyzer. strategyzer.com
Desk research	Research that draws on material which already exists, such as published studies, policy papers, usage data and support records, before any new data is collected.	R04Discovery researchFrom the field · not in the guide	Hall, E. (2019). Just enough research (2nd ed.). A Book Apart. abookapart.com
Desktop walkthrough	A service prototype in which a team acts out a journey on a tabletop with small figures and a simple map, to find gaps before anything is built.	R02Service design and systems thinkingFrom the field · not in the guide	Stickdorn, M., Hormess, M. E., Lawrence, A., & Schneider, J. (2018). This is service design doing: Applying service design thinking in the real world. O'Reilly Media.
DevOps	A movement of developers and operations staff working together, with automation, feature flags, shared measures and a culture that avoids blame, to make releasing so ordinary that it stops being dangerous (Allspaw & Hammond, 2009).	R12DevOps and continuous deliveryThe misconception · slide 10 (opens in a new tab)	Allspaw, J., & Hammond, P. (2009). 10+ deploys per day: Dev and ops cooperation at Flickr [Conference presentation]. O'Reilly Velocity Conference 2009, San Jose, CA, United States. slideshare.net Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com devopsdays. (n.d.). About devopsdays. Retrieved September 17, 2026, from devopsdays.org
Diary study	A study in which participants record their own activities, thoughts or problems over days or weeks, capturing events a researcher could not be present for.	R04Discovery researchFrom the field · not in the guide	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: Capturing life as it is lived. Annual Review of Psychology, 54, 579–616. doi
Difference-in-differences	A quasi-experimental method that compares the change over time in a group that received an intervention with the change in a group that did not.	R13Experimentation and product analyticsFrom the field · not in the guide	Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. doi
Differential privacy	A mathematical guarantee that a published statistic changes very little whether or not any one person's data is included, achieved by adding calibrated random noise.	R17Privacy, security and ethicsFrom the field · not in the guide	Dwork, C., McSherry, F., Nissim, K., & Smith, A. (2006). Calibrating noise to sensitivity in private data analysis. In S. Halevi & T. Rabin (Eds.), Theory of cryptography (pp. 265–284). Springer. doi Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3–4), 211–407. doi
Diffusion	The process by which an innovation is communicated through certain channels over time among the members of a social system (Rogers, 2003).	R19Adoption and change in schoolsThe reluctant adopter · slide 27 (opens in a new tab)	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press. Ryan, B., & Gross, N. C. (1943). The diffusion of hybrid seed corn in two Iowa communities. Rural Sociology, 8(1), 15–24.
Direction	Feedback that tells a designer what to change, usually without a reason tied to a goal.	R07Design methods IICritique against the goal · slide 32 (opens in a new tab)	Connor, A. (2016). 3 kinds of feedback. Discussing Design, Medium. medium.com Connor, A., & Irizarry, A. (2015). Discussing design: Improving communication and collaboration through critique. O'Reilly Media. oreilly.com
Discovery	The work of deciding whether something is worth building, using quick and cheap tests.	R01Digital product managementThe spine · slide 15 (opens in a new tab)	Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley. Torres, T. (2021). Continuous discovery habits: Discover products that create customer value and business value. Product Talk. Sy, D. (2007). Adapting usability investigations for agile user-centered design. Journal of Usability Studies, 2(3), 112–132. uxpajournal.org
Discovery research	Research done before designing, to learn what people do now and why.	R04Discovery researchThe spine · slide 12 (opens in a new tab)	Hall, E. (2019). Just enough research (2nd ed.). A Book Apart. abookapart.com Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com
Discussion guide	The interviewer's plan for a session: the topics, the main questions and rough timings. It is a prompt to work from, not a script to read out.	R04Discovery researchFrom the field · not in the guide	Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com Hall, E. (2019). Just enough research (2nd ed.). A Book Apart. abookapart.com
Disparate impact	Harm that falls more heavily on a protected group from a rule or model that is neutral on its face, whatever the intent behind it.	R17Privacy, security and ethicsFrom the field · not in the guide	Griggs v. Duke Power Co., 401 U.S. 424 (1971). Barocas, S., & Selbst, A. D. (2016). Big data's disparate impact. California Law Review, 104(3), 671–732.
Distributed leadership	Leadership spread across several people, such as heads of department and subject leads, which is how a tool becomes part of everyday routines (Leithwood et al., 2008).	R19Adoption and change in schoolsLeader, then department · slide 34 (opens in a new tab)	Leithwood, K., Harris, A., & Hopkins, D. (2008). Seven strong claims about successful school leadership. School Leadership & Management, 28(1), 27–42. doi
Distribution	The pattern of how values spread out: where they cluster, how far they range and which way they lean.	R18Basic statistics for educational technologistsWhat statistics is for · slide 11 (opens in a new tab)	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Kunin, D., Guo, J., Devlin, T. D., & Xiang, D. (n.d.). Seeing theory: A visual introduction to probability and statistics. Brown University. seeing-theory.brown.edu
Divergent thinking	Widening the range of options or perspectives before choosing among them.	R03Design thinking, used honestlyThe spine · slide 14 (opens in a new tab)	Design Council. (n.d.-a). Framework for innovation. Retrieved September 17, 2026, from designcouncil.org.uk Design Council. (2007). Eleven lessons: Managing design in eleven global brands. A study of the design process. idi-design.ie
Domain Name System (DNS)	The internet's directory, which turns a name such as a school's web address into the numeric address of the server to contact. A DNS fault makes a working service unreachable.	R11Software development for non-engineersFrom the field · not in the guide	MDN Web Docs. (n.d.). How the web works. Mozilla. Retrieved September 17, 2026, from developer.mozilla.org Mockapetris, P. (1987). Domain names: Concepts and facilities (RFC 1034). Internet Engineering Task Force. doi
DORA measures	Five measures of software delivery performance, three of throughput and two of instability, read together and never as targets (DORA, 2026).	R12DevOps and continuous deliveryThe dora measures · slide 37 (opens in a new tab)	DORA. (2026). DORA's software delivery performance metrics. Google Cloud. dora.dev Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com Harvey, N. (2026). A history of DORA's software delivery metrics. DORA. dora.dev
Dosage	The amount of a programme a participant actually receives, counted in hours, sessions or coaching cycles, as distinct from the amount planned.	R21Professional learning that changes practiceFrom the field · not in the guide
Dot voting	A quick way for a group to narrow options: each person gets a few sticky dots and places them on the ideas they favour.	R07Design methods IIFrom the field · not in the guide	Gray, D., Brown, S., & Macanufo, J. (2010). Gamestorming: A playbook for innovators, rulebreakers, and changemakers. O'Reilly Media. Knapp, J., Zeratsky, J., & Kowitz, B. (2016). Sprint: How to solve big problems and test new ideas in just five days. Simon & Schuster.
Double Diamond	The Design Council's model of design as two rounds of divergence and convergence: discover and define, then develop and deliver.	R03Design thinking, used honestlyThe spine · slide 14 (opens in a new tab)	Design Council. (2007). Eleven lessons: Managing design in eleven global brands. A study of the design process. idi-design.ie Design Council. (n.d.-b). History of the Double Diamond. Retrieved September 17, 2026, from designcouncil.org.uk Design Council. (n.d.-a). Framework for innovation. Retrieved September 17, 2026, from designcouncil.org.uk
Double-barrelled question	A question that asks about two things at once, such as whether a tool is "quick and easy", so an answer cannot be tied to either.	R05Evaluative researchFrom the field · not in the guide	Pew Research Center. (2021). Writing survey questions. pewresearch.org
Driver diagram	A one-page picture of an improvement team's working theory: the aim, the primary and secondary drivers believed to produce it, and the change ideas to be tested against each.	R10Lean product developmentFrom the field · not in the guide	Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America's schools can get better at getting better. Harvard Education Press. books.google.com
Dual-track development	Running discovery and delivery as two parallel tracks, two kinds of work and two kinds of thinking, with ideas tested and settled ahead of the code that implements them (Patton, 2017).	R01Digital product managementThe spine · slide 15 (opens in a new tab)	Sy, D. (2007). Adapting usability investigations for agile user-centered design. Journal of Usability Studies, 2(3), 112–132. uxpajournal.org Patton, J. (2017, May 10). Dual track development is not duel track. Jeff Patton & Associates. jpattonassociates.com Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley.
E
Early-warning system	A tool that predicts which students may fail or leave, so that support can be offered early.	R15Learning analyticsShow the uncertainty · slide 33 (opens in a new tab)	Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. doi Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi Perdomo, J. C., Britton, T., Hardt, M., & Abebe, R. (2025). Difficult lessons on social prediction from Wisconsin public schools. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (pp. 2682–2704). ACM. doi
Economic principle	The best prototype makes an idea's possibilities and limits visible in the simplest, most efficient way (Lim et al., 2008).	R07Design methods IIThe spine · slide 16 (opens in a new tab)	Lim, Y.-K., Stolterman, E., & Tenenberg, J. (2008). The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Transactions on Computer-Human Interaction, 15(2), Article 7. doi Rudd, J., Stern, K., & Isensee, S. (1996). Low vs. high-fidelity prototyping debate. Interactions, 3(1), 76–85. doi
Ecosystem map	A picture of every person, organisation, system and rule around a user's task.	R02Service design and systems thinkingBeyond your screen · slide 24 (opens in a new tab)	Polaine, A., Løvlie, L., & Reason, B. (2013). Service design: From insight to implementation. Rosenfeld Media. Stickdorn, M., Hormess, M. E., Lawrence, A., & Schneider, J. (2018). This is service design doing: Applying service design thinking in the real world. O'Reilly Media. Kalbach, J. (2020). Mapping experiences: A complete guide to customer alignment through journeys, blueprints, and diagrams (2nd ed.). O'Reilly Media. openlibrary.org
Ed-Fi Data Standard	A standard covering K–12 information about students and their academic performance, used by a number of US states and districts, for example for records sent to the state (Ed-Fi Alliance, n.d.).	R14Data strategy and governanceIdentifiers first · slide 39 (opens in a new tab)	Ed-Fi Alliance. (n.d.). About the Ed-Fi Data Standard. Retrieved September 17, 2026, from docs.ed-fi.org
Educational data mining (EDM)	A sister field to learning analytics that develops methods for finding patterns in educational data, with an emphasis on automated discovery and modelling.	R15Learning analyticsFrom the field · not in the guide	Baker, R. S. J. d., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17. doi Siemens, G., & Baker, R. S. J. d. (2012). Learning analytics and educational data mining: Towards communication and collaboration. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 252–254). ACM. doi
Effect size	The size of a difference between groups, often in standard deviations. Comparable only when tests and comparison groups are similar. R18 The size of a difference or relationship, in raw units or standardised against the spread. R21 A difference between groups expressed in standard deviations, so results from different tests can be compared.	R16AI in learning productsDate the evidence · slide 40 (opens in a new tab) R18Basic statistics for educational technologistsEffect size · slide 30 (opens in a new tab) R21Professional learning that changes practiceCoaching · slide 33 (opens in a new tab)	von Hippel, P. T. (2024). Two-sigma tutoring: Separating science fiction from science fact. Education Next, 24(2). educationnext.org Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. Kraft, M. A., Blazar, D., & Hogan, D. (2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research, 88(4), 547–588. doi
Effectiveness trial	A test of a programme at larger scale under ordinary conditions.	R21Professional learning that changes practiceCoaching · slide 34 (opens in a new tab)	Kraft, M. A., Blazar, D., & Hogan, D. (2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research, 88(4), 547–588. doi
Efficacy trial	A test of a programme under favourable conditions, usually small and closely supported.	R21Professional learning that changes practiceCoaching · slide 34 (opens in a new tab)	Kraft, M. A., Blazar, D., & Hogan, D. (2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research, 88(4), 547–588. doi
Effort ledger	A count of the minutes a change adds and removes for one person over a set period.	R19Adoption and change in schoolsEffort before training · slide 20 (opens in a new tab)	Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. doi Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. doi
Eight-step change model	Kotter's sequence for leading change, which runs from creating urgency and building a guiding coalition, through short-term wins, to anchoring the change in the culture.	R19Adoption and change in schoolsFrom the field · not in the guide	Kotter, J. P. (1996). Leading change. Harvard Business School Press. doi Kotter, J. P. (1995). Leading change: Why transformation efforts fail. Harvard Business Review, 73(2), 59–67.
Embargo	An agreement that information shared in advance will not be published or passed on before a set date and time.	R20Communicating change to schoolsFrom the field · not in the guide
Embedding	A list of numbers that represents a piece of text, an image or another item so that items with similar meaning sit close together. It is the basis of semantic search.	R16AI in learning productsFrom the field · not in the guide	Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1532–1543. doi
Empathy	In design, the effort to understand a situation as the people in it experience it, gained by observing, talking with and working alongside them, not by imagining them.	R03Design thinking, used honestlyFrom the field · not in the guide	Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92. hbr.org Kouprie, M., & Sleeswijk Visser, F. (2009). A framework for empathy in design: Stepping into and out of the user's life. Journal of Engineering Design, 20(5), 437–448. doi Leonard, D., & Rayport, J. F. (1997). Spark innovation through empathic design. Harvard Business Review, 75(6), 102–113.
Empathy map	A one-page chart of what a type of user says, thinks, does and feels, used by a team to pool what it knows about that user.	R06Design methods IFrom the field · not in the guide	Gray, D., Brown, S., & Macanufo, J. (2010). Gamestorming: A playbook for innovators, rulebreakers, and changemakers. O'Reilly Media.
Empiricism	The view that knowledge comes from experience and from making decisions based on what is observed; the foundation on which Scrum rests (Schwaber & Sutherland, 2020).	R09Agile and Scrum in practiceThe spine · slide 16 (opens in a new tab)	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Empowered product team	A cross-functional team that is given a problem and an outcome to achieve, and the authority to decide the solution, in place of a list of features to build.	R01Digital product managementFrom the field · not in the guide	Cagan, M., Hickman, L., Jones, C., Idiodi, C., & Moore, J. (2024). Transformed: Moving to the product operating model. Wiley. Cagan, M., & Jones, C. (2021). Empowered: Ordinary people, extraordinary products. Wiley.
Encryption at rest	Encrypting stored data, in databases, backups and laptops, so that a stolen disk or copied file cannot be read without the key.	R17Privacy, security and ethicsFrom the field · not in the guide	Scarfone, K., Souppaya, M., & Sexton, M. (2007). Guide to storage encryption technologies for end user devices (NIST Special Publication 800-111). National Institute of Standards and Technology. doi Anderson, R. (2020). Security engineering: A guide to building dependable distributed systems (3rd ed.). Wiley. doi
Encryption in transit	Encrypting data as it moves between a device and a server or between servers, usually with TLS, so that it cannot be read or altered on the way.	R17Privacy, security and ethicsFrom the field · not in the guide	Rescorla, E. (2018). The Transport Layer Security (TLS) protocol version 1.3 (RFC 8446). Internet Engineering Task Force. doi
End of life (EOL)	The date after which a product or version is no longer sold, updated or supported, announced in advance so users can move off it.	R20Communicating change to schoolsFrom the field · not in the guide
End-to-end service	A service considered from the moment a user first has a need to the moment it is fully met, across every channel and organisation involved, not only the part one team owns.	R02Service design and systems thinkingFrom the field · not in the guide	Downe, L. (2020). Good services: How to design services that work. BIS Publishers. good.services Government Digital Service. (2019). Service Standard. GOV.UK. gov.uk
End-to-end test	An automated test that drives the whole running system as a user would, for example by logging in through a browser and submitting work. Slow and fragile, so kept few.	R11Software development for non-engineersFrom the field · not in the guide	Fowler, M. (2012, May 1). Test pyramid. martinfowler.com. martinfowler.com
Epic	A user story too large to finish in one sprint, kept as a placeholder and split into smaller stories before work starts.	R09Agile and Scrum in practiceFrom the field · not in the guide	Cohn, M. (2004). User stories applied: For agile software development. Addison-Wesley. mountaingoatsoftware.com Cohn, M. (2005). Agile estimating and planning. Prentice Hall.
Error budget	The amount of failure a reliability target allows over a period; one minus the target. R12 The amount of unreliability a service may have in a period before releases pause (Beyer et al., 2016).	R11Software development for non-engineersKnowing it is broken · slide 38 (opens in a new tab) R12DevOps and continuous deliveryFreeze by exception · slide 45 (opens in a new tab)	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Error rate	The number of mistakes users make while attempting a task, reported per task or per opportunity for error.	R05Evaluative researchFrom the field · not in the guide	Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann. shop.elsevier.com
Escalation	Passing a ticket to someone with more expertise (functional escalation) or more authority (hierarchical escalation) when it cannot be resolved where it is.	R22Support and feedback loopsFrom the field · not in the guide
Ethics review board	A committee that reviews research involving people before it starts, weighing risks, benefits, consent and fairness. It is called an institutional review board (IRB) in the United States.	R17Privacy, security and ethicsFrom the field · not in the guide	Protection of Human Subjects, 45 C.F.R. Part 46 (2018). National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (1979). The Belmont report: Ethical principles and guidelines for the protection of human subjects of research. U.S. Department of Health, Education, and Welfare.
Ethnography	The study of a group in its own setting over an extended period, through observation and taking part, to understand practices as the group itself understands them.	R04Discovery researchFrom the field · not in the guide	Hammersley, M., & Atkinson, P. (2019). Ethnography: Principles in practice (4th ed.). Routledge. doi
EU AI Act	The European Union's 2024 regulation laying down harmonised rules on artificial intelligence, which lists systems used to evaluate learning outcomes among high-risk uses (European Union, 2024).	R16AI in learning productsA human accountable · slide 36 (opens in a new tab)	European Union. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L 2024/1689. eur-lex.europa.eu Council of the European Union. (2026, June 29). Artificial intelligence: Council gives final green light to simplify and streamline rules [Press release]. consilium.europa.eu
Evaluation set	A fixed collection of real tasks with agreed answers, used to test a feature each time the model or prompt changes.	R16AI in learning productsEvaluate first · slide 19 (opens in a new tab)	National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. doi Department for Education. (2026). Generative AI: Product safety standards (Updated January 19, 2026). GOV.UK. gov.uk
Evaluator effect	The finding that different evaluators using the same method on the same system report different problems (Hertzum & Jacobsen, 2001).	R05Evaluative researchThe misconception · slide 9 (opens in a new tab)	Hertzum, M., & Jacobsen, N. E. (2001). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human–Computer Interaction, 13(4), 421–443. doi
Evidence trail	The links from an insight or decision back to the observations it rests on.	R06Design methods IKeep the trail · slide 26 (opens in a new tab)	Pidcock, D. (2018). What is atomic UX research? Prototypr. blog.prototypr.io Sharon, T. (2018). Foundations of atomic research. Medium. tsharon.medium.com
Evolutionary prototype	A prototype built to be kept and refined, round by round, until it becomes the product.	R07Design methods IIFrom the field · not in the guide	Tripp, S. D., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative instructional design strategy. Educational Technology Research and Development, 38(1), 31–44. doi Floyd, C. (1984). A systematic look at prototyping. In R. Budde, K. Kuhlenkamp, L. Mathiassen, & H. Züllighoven (Eds.), Approaches to prototyping (pp. 1–18). Springer. doi
Exemplar	A strong piece of finished work shown to learners before they begin, so they can see and discuss what quality looks like.	R07Design methods IIFrom the field · not in the guide	Berger, R. (2003). An ethic of excellence: Building a culture of craftsmanship with students. Heinemann. heinemann.com Berger, R., Rugen, L., & Woodfin, L. (2014). Leaders of their own learning: Transforming schools through student-engaged assessment. Jossey-Bass. wiley.com
Exit and voice	Two responses to a failing organisation: leaving it, or speaking up to change it (Hirschman, 1970).	R22Support and feedback loopsThe spine · slide 16 (opens in a new tab)	Hirschman, A. O. (1970). Exit, voice, and loyalty: Responses to decline in firms, organizations, and states. Harvard University Press. hup.harvard.edu
Experience API (xAPI)	A specification for recording what learners did and sending it to a learning record store, published as IEEE 9274.1.1-2023 (IEEE, 2023).	R14Data strategy and governanceIdentifiers first · slide 39 (opens in a new tab)	IEEE. (2023). IEEE standard for learning technology: JavaScript Object Notation (JSON) data model format and Representational State Transfer (RESTful) web service for learner experience data tracking and access (IEEE 9274.1.1-2023). standards.ieee.org
Experience prototyping	Any representation, in any medium, that lets designers, users or clients go through a future situation themselves instead of being told about it.	R07Design methods IIFrom the field · not in the guide	Buchenau, M., & Fulton Suri, J. (2000). Experience prototyping. In Proceedings of the 3rd Conference on Designing Interactive Systems (pp. 424–433). ACM. doi
Explain-back	A check in which the people affected describe the change and what they will do, in their own words.	R20Communicating change to schoolsEquip the middle · slide 46 (opens in a new tab)	General Services Administration. (n.d.). Paraphrase testing. Digital.gov plain language guide series. Retrieved September 17, 2026, from digital.gov
Explainability	How far a system can give reasons for an output that the people affected can understand and check.	R16AI in learning productsFrom the field · not in the guide	National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework (AI RMF 1.0) (NIST AI 100-1). U.S. Department of Commerce. doi Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022). Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence, 3, Article 100074. doi
Explainer pack	A short set of materials that lets a school leader or head of department explain a change.	R20Communicating change to schoolsEquip the middle · slide 45 (opens in a new tab)	Coburn, C. E. (2005). Shaping teacher sensemaking: School leaders and the enactment of reading policy. Educational Policy, 19(3), 476–509. doi Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation (3rd ed.). Education Endowment Foundation. educationendowmentfoundation.org.uk
External validity	The extent to which a study's result holds for other people, settings, times and versions of the intervention than those studied.	R13Experimentation and product analyticsFrom the field · not in the guide	Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Extract, transform, load (ETL)	The process of copying data out of source systems, reshaping and cleaning it, and loading it into a warehouse or other target store.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Kimball, R., & Ross, M. (2013). The data warehouse toolkit: The definitive guide to dimensional modeling (3rd ed.). Wiley.
Extraction	Taking participants' time and ideas without giving anything back, including an account of what their input changed.	R03Design thinking, used honestlyProtect their time · slide 43 (opens in a new tab)	Penuel, W. R., Roschelle, J., & Shechtman, N. (2007). Designing formative assessment software with teachers: An analysis of the co-design process. Research and Practice in Technology Enhanced Learning, 2(1), 51–74. doi Iskander, N. (2018). Design thinking is fundamentally conservative and preserves the status quo. Harvard Business Review. hbr.org
Extraneous load	Load caused by how information is presented, rather than by its complexity (Sweller et al., 2019).	R08UX foundations and design systemsThe spine · slide 15 (opens in a new tab)	Sweller, J., van Merriënboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–296. doi Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31(2), 261–292. doi Mayer, R. E. (2024). The past, present, and future of the cognitive theory of multimedia learning. Educational Psychology Review, 36(1), Article 8. doi
Extreme Programming (XP)	An agile method built on engineering practices such as pair programming, test-first development, continuous integration and small, frequent releases.	R09Agile and Scrum in practiceFrom the field · not in the guide	Beck, K., & Andres, C. (2004). Extreme programming explained: Embrace change (2nd ed.). Addison-Wesley.
Eye tracking	Recording where on a screen a person looks, for how long and in what order, using a device that follows the movement of their eyes.	R05Evaluative researchFrom the field · not in the guide	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com Nielsen, J., & Pernice, K. (2010). Eyetracking web usability. New Riders.
F
F-shaped pattern	The way people often scan web text: across the top, across again a little lower, then down the left side (Pernice, 2017).	R20Communicating change to schoolsLead with the teacher · slide 20 (opens in a new tab)	Pernice, K. (2017). F-shaped pattern of reading on the web: Misunderstood, but still relevant (even on mobile). Nielsen Norman Group. nngroup.com Nielsen, J. (1997). How users read on the web. Nielsen Norman Group. nngroup.com Nielsen, J. (2008). How little do users read? Nielsen Norman Group. nngroup.com
Facilitating conditions	The support and infrastructure a person believes exist for using a system (Venkatesh et al., 2003).	R19Adoption and change in schoolsEffort before training · slide 21 (opens in a new tab)	Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. doi Venkatesh, V., Thong, J. Y. L., & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly, 36(1), 157–178. doi
Failed deployment recovery time	How long it takes to recover from a deployment that fails and needs immediate intervention (DORA, 2026). R22 The time it takes to recover from a deployment that fails and requires immediate intervention (DORA, 2026).	R12DevOps and continuous deliveryThe dora measures · slide 37 (opens in a new tab) R22Support and feedback loopsTime to fix · slide 41 (opens in a new tab)	DORA. (2026). DORA's software delivery performance metrics. Google Cloud. dora.dev Harvey, N. (2026). A history of DORA's software delivery metrics. DORA. dora.dev Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution Press.
Failure demand	Demand on a service caused by a failure to do something, or to do it right, for the user (Seddon, 2003).	R02Service design and systems thinkingClose the loop · slide 43 (opens in a new tab) R22Support and feedback loopsThe misconception · slide 11 (opens in a new tab)	Seddon, J. (2003). Freedom from command and control: A better way to make the work work. Vanguard Education.
FAIR principles	Guidelines that data should be findable, accessible, interoperable and reusable, by machines as well as people. FAIR data need not be open data.	R14Data strategy and governanceFrom the field · not in the guide	Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., Bouwman, J., Brookes, A. J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C. T., Finkers, R., . . . Mons, B. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. doi
Fairness metric	A statistic that compares a model's behaviour across groups, such as selection rates or error rates. The common metrics cannot all be met at once, so one must be chosen and justified.	R17Privacy, security and ethicsFrom the field · not in the guide	Barocas, S., Hardt, M., & Narayanan, A. (2023). Fairness and machine learning: Limitations and opportunities. MIT Press. Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2), 153–163. doi
Fake-door test	Offering a feature that does not yet exist and counting who tries to use it.	R10Lean product developmentThe smallest test · slide 26 (opens in a new tab)	Bland, D. J., & Osterwalder, A. (2019). Testing business ideas: A field guide for rapid experimentation. Wiley. books.google.com
False alarm	A student flagged as at risk who would have been fine. Also called a false positive.	R15Learning analyticsThe base rate · slide 39 (opens in a new tab)	Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi Feathers, T. (2023). False alarm: How Wisconsin uses race and income to label students “high risk”. The Markup. themarkup.org
Family Educational Rights and Privacy Act (FERPA)	The 1974 United States law that gives parents, and students from 18, rights over education records, and limits what a school receiving federal funds may disclose without consent.	R17Privacy, security and ethicsFrom the field · not in the guide	Family Educational Rights and Privacy Act of 1974, 20 U.S.C. § 1232g. Family Educational Rights and Privacy, 34 C.F.R. Part 99.
Feature flag	A setting that switches a feature on or off, or for some users only, without changing code (Hodgson, 2017).	R12DevOps and continuous deliveryDeploy is not release · slide 31 (opens in a new tab)	Hodgson, P. (2017). Feature toggles (aka feature flags). martinfowler.com. martinfowler.com Allspaw, J., & Hammond, P. (2009). 10+ deploys per day: Dev and ops cooperation at Flickr [Conference presentation]. O'Reilly Velocity Conference 2009, San Jose, CA, United States. slideshare.net
Feature request	A user's suggestion for something the product does not yet do, logged separately from faults and passed to the product team.	R22Support and feedback loopsFrom the field · not in the guide
Federated model	A model in which the teams closest to data own it, and a central group sets standards they follow. R08 A way of owning a design system in which a representative, empowered subset of designers and leaders from product teams shape it together, with a central group to document and maintain it (Curtis, 2015).	R14Data strategy and governanceNamed owners · slide 26 (opens in a new tab) R08UX foundations and design systemsRun it as a product · slide 43 (opens in a new tab)	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Curtis, N. (2015). Team models for scaling a design system. EightShapes. medium.com Dehghani, Z. (2022). Data mesh: Delivering data-driven value at scale. O'Reilly Media.
Feedback loop	A route by which the effects of an action return to influence the next action.	R02Service design and systems thinkingClose the loop · slide 42 (opens in a new tab)	Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green. Sterman, J. D. (1989). Modeling managerial behavior: Misperceptions of feedback in a dynamic decision making experiment. Management Science, 35(3), 321–339. doi Sterman, J. D. (2006). Learning from evidence in a complex world. American Journal of Public Health, 96(3), 505–514. doi
Fidelity	How close a prototype is to the finished product in look, behaviour and content. R21 The degree to which a programme is delivered as designed.	R07Design methods IIMatch the fidelity · slide 25 (opens in a new tab) R21Professional learning that changes practiceThe second week · slide 49 (opens in a new tab)	Rudd, J., Stern, K., & Isensee, S. (1996). Low vs. high-fidelity prototyping debate. Interactions, 3(1), 76–85. doi Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov Walker, M., Takayama, L., & Landay, J. A. (2002). High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 46(5), 661–665. doi
Field notes	The researcher's written record of what was seen and heard during fieldwork, made at the time or soon after, keeping description apart from the researcher's own reading of it.	R04Discovery researchFrom the field · not in the guide	Emerson, R. M., Fretz, R. I., & Shaw, L. L. (2011). Writing ethnographic fieldnotes (2nd ed.). University of Chicago Press. doi
Filters and manifestations	Two roles of a prototype: a filter selects the aspects of an idea to examine, and a manifestation makes the idea concrete in some material, level of detail and scope (Lim et al., 2008).	R07Design methods IIThe spine · slide 16 (opens in a new tab)	Lim, Y.-K., Stolterman, E., & Tenenberg, J. (2008). The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Transactions on Computer-Human Interaction, 15(2), Article 7. doi
Finding	A statement of what the research observed, backed by evidence. It comes before an insight, which says what the finding means and why it matters.	R06Design methods IFrom the field · not in the guide	Pidcock, D. (2018). What is atomic UX research? Prototypr. blog.prototypr.io Sharon, T. (2018). Foundations of atomic research. Medium. tsharon.medium.com
Fine-tuning	Further training of an already-trained model on a smaller, specific dataset so that it performs better on a particular task or in a particular style.	R16AI in learning productsFrom the field · not in the guide	Howard, J., & Ruder, S. (2018). Universal language model fine-tuning for text classification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 328–339. doi
First response time	The time between a user raising a ticket and the first reply from a person, not counting automatic acknowledgements.	R22Support and feedback loopsFrom the field · not in the guide
First-contact resolution (FCR)	The share of requests resolved in the user's first contact, with no follow-up, transfer or repeat contact needed.	R22Support and feedback loopsFrom the field · not in the guide
First-order barrier	An external obstacle to using technology, such as time, access or support (Ertmer, 1999).	R19Adoption and change in schoolsLeader, then department · slide 33 (opens in a new tab)	Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47(4), 47–61. doi
Fitts's law	The time to reach a target depends on its size and distance, so a large, clearly labelled primary action is easier to hit (Fitts, 1954).	R08UX foundations and design systemsThe next step · slide 25 (opens in a new tab)	Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47(6), 381–391. doi Budiu, R. (2022). Fitts's law and its applications in UX. Nielsen Norman Group. nngroup.com
Five levels of evaluation	Five levels of evidence for professional learning: participants' reactions, their learning, organisation support and change, their use of new knowledge and skills, and student learning outcomes (Guskey, 2002).	R21Professional learning that changes practiceMeasure practice · slide 40 (opens in a new tab)	Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45–51. ascd.org Guskey, T. R. (2000). Evaluating professional development. Corwin Press.
Five whys	Asking 'why?' of a problem repeatedly, about five times, to get past its symptoms to a cause worth fixing.	R10Lean product developmentFrom the field · not in the guide	Ohno, T. (1988). Toyota production system: Beyond large-scale production. Productivity Press. Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com
Five-step model	The basic, linear process of empathise, define, ideate, prototype and test popularised by Stanford's d.school, which its academic director calls just a first recipe, a suggestion for how to get started (Carter, 2016).	R03Design thinking, used honestlyThe misconception · slide 8 (opens in a new tab)	Carter, C. (2016). Let's stop talking about THE design process. Stanford d.school. dschool.stanford.edu Stanford d.school. (2025). Let's stop talking about THE design process [Introduction to Carter, 2016]. dschool.stanford.edu Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92. hbr.org
Five-user rule	The advice to test with five users, from a model in which one user reveals about 31% of problems, so five find about 85% on average, though any five may not (Nielsen, 2000).	R05Evaluative researchSmall and often · slide 25 (opens in a new tab)	Nielsen, J., & Landauer, T. K. (1993). A mathematical model of the finding of usability problems. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (pp. 206–213). ACM. doi Nielsen, J. (2000). Why you only need to test with 5 users. Nielsen Norman Group. nngroup.com Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 379–383. doi
Flaky test	An automated test that sometimes passes and sometimes fails on the same code. Flaky tests teach a team to ignore failures, which undermines the pipeline.	R12DevOps and continuous deliveryFrom the field · not in the guide	Luo, Q., Hariri, F., Eloussi, L., & Marinov, D. (2014). An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 643–653). ACM. doi
Flow efficiency	The share of an item's total elapsed time during which someone is actually working on it, as opposed to it waiting in a queue.	R10Lean product developmentFrom the field · not in the guide	Modig, N., & Åhlström, P. (2012). This is lean: Resolving the efficiency paradox. Rheologica Publishing.
Focus group	A moderated discussion among a small group of participants on a set topic. It shows how people talk about a subject together, not how each of them behaves.	R04Discovery researchFrom the field · not in the guide	Krueger, R. A., & Casey, M. A. (2015). Focus groups: A practical guide for applied research (5th ed.). SAGE.
Forgetting curve	The pattern of rapid early loss of learned material, first described by Ebbinghaus and since replicated (Murre & Dros, 2015).	R21Professional learning that changes practiceThe spine · slide 14 (opens in a new tab)	Murre, J. M. J., & Dros, J. (2015). Replication and analysis of Ebbinghaus' forgetting curve. PLOS ONE, 10(7), Article e0120644. doi Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380. doi
Formative and summative testing	Formative testing looks for problems to fix while a design is still changing. Summative testing measures how well a finished design performs, often against a target or an earlier version.	R05Evaluative researchFrom the field · not in the guide	Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann. shop.elsevier.com Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Wiley.
Foundation model	A large model trained on broad data that can be adapted to many tasks, and on which many separate products are built.	R16AI in learning productsFrom the field · not in the guide	National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. doi Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., … Liang, P. (2021). On the opportunities and risks of foundation models [Preprint]. arXiv. doi
Four golden signals	Four signals worth watching on any live service: latency, traffic, errors and saturation (Beyer et al., 2016).	R11Software development for non-engineersKnowing it is broken · slide 38 (opens in a new tab)	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Frame	A way of seeing a problem that implies how value could be created (Dorst, 2011).	R03Design thinking, used honestlyThe spine · slide 16 (opens in a new tab)	Dorst, K. (2011). The core of 'design thinking' and its application. Design Studies, 32(6), 521–532. doi Dorst, K. (2015). Frame innovation: Create new thinking by design. MIT Press. mitpress.mit.edu Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Basic Books.
Framework method	A way of analysing qualitative data by summarising it into a matrix of cases against themes, so that it can be read across cases and within each case.	R06Design methods IFrom the field · not in the guide	Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013). Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Medical Research Methodology, 13, Article 117. doi
Freemium	A pricing model in which a basic version of a product is free and users or institutions pay for further features, capacity or support.	R01Digital product managementFrom the field · not in the guide	Anderson, C. (2009). Free: The future of a radical price. Hyperion.
Front end	The early, uncertain stage of design where a team decides what to design, if anything (Sanders & Stappers, 2008). R11 The part of a system that runs in the browser or app and that users see and touch.	R03Design thinking, used honestlyResearch first · slide 39 (opens in a new tab) R11Software development for non-engineersThe spine · slide 15 (opens in a new tab)	Sanders, E. B.-N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. CoDesign, 4(1), 5–18. doi MDN Web Docs. (n.d.). How the web works. Mozilla. Retrieved September 17, 2026, from developer.mozilla.org
Front-loading	Putting the most important information at the start of a message, heading or line.	R20Communicating change to schoolsLead with the teacher · slide 20 (opens in a new tab)	Pernice, K. (2017). F-shaped pattern of reading on the web: Misunderstood, but still relevant (even on mobile). Nielsen Norman Group. nngroup.com Government Digital Service. (n.d.-c). Create a clear structure for your content. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk Nielsen, J. (1997). How users read on the web. Nielsen Norman Group. nngroup.com
Frontstage	Work and touchpoints the user sees directly.	R02Service design and systems thinkingMap the backstage · slide 19 (opens in a new tab)	Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008). Service blueprinting: A practical technique for service innovation. California Management Review, 50(3), 66–94. doi Shostack, G. L. (1984). Designing services that deliver. Harvard Business Review, 62(1), 133–139. hbr.org Gibbons, S. (2017, August 27). Service blueprints: Definition. Nielsen Norman Group. nngroup.com
Function creep	The gradual use of data or a system for purposes beyond the one it was set up for, such as attendance data reused for discipline.	R17Privacy, security and ethicsFrom the field · not in the guide	Koops, B.-J. (2021). The concept of function creep. Law, Innovation and Technology, 13(1), 29–56. doi
G
Gallery critique	A classroom protocol in which work is displayed and peers give kind, specific and helpful comments.	R07Design methods IICritique against the goal · slide 33 (opens in a new tab)	Berger, R. (2003). An ethic of excellence: Building a culture of craftsmanship with students. Heinemann. heinemann.com Berger, R., Rugen, L., & Woodfin, L. (2014). Leaders of their own learning: Transforming schools through student-engaged assessment. Jossey-Bass. wiley.com Sherrington, T., & Stafford, S. (2019). Effective feedback: Class critique. Chartered College of Teaching. my.chartered.college
Gatekeeper	A person who controls access to participants, such as a school leader.	R04Discovery researchWatch first · slide 21 (opens in a new tab)	British Educational Research Association. (2024). Ethical guidelines for educational research (5th ed.). bera.ac.uk
Gemba walk	A visit by managers to the place where the work is done, to watch it and ask questions there instead of relying on reports. Gemba is Japanese for 'the actual place'.	R10Lean product developmentFrom the field · not in the guide	Imai, M. (1997). Gemba kaizen: A commonsense, low-cost approach to management. McGraw-Hill. Liker, J. K. (2004). The Toyota way: 14 management principles from the world's greatest manufacturer. McGraw-Hill.
General Data Protection Regulation (GDPR)	The European Union's data protection law, in force since 2018. It applies to anyone who offers services to, or monitors, people in the Union, and is the model for many later laws.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88.
Generative AI	AI that produces new text, images, audio or code in response to a prompt, by modelling patterns in the data it was trained on.	R16AI in learning productsFrom the field · not in the guide	Miao, F., & Holmes, W. (2023). Guidance for generative AI in education and research. UNESCO. unesco.org National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. doi
Gestalt principles	Rules describing how people see separate visual elements as groups, such as proximity, similarity and closure. Designers use them to show which parts of a screen belong together.	R08UX foundations and design systemsFrom the field · not in the guide	Wertheimer, M. (1923). Untersuchungen zur Lehre von der Gestalt. II. Psychologische Forschung, 4(1), 301–350. doi Johnson, J. (2020). Designing with the mind in mind: Simple guide to understanding user interface design guidelines (3rd ed.). Morgan Kaufmann. doi
Git	The most widely used version control system. Each developer holds a full copy of the project's history and exchanges changes with a shared copy, usually on a hosting service.	R11Software development for non-engineersFrom the field · not in the guide	Chacon, S., & Straub, B. (2014). Pro Git (2nd ed.). Apress. Loeliger, J., & McCullough, M. (2012). Version control with Git: Powerful tools and techniques for collaborative software development (2nd ed.). O'Reilly Media.
Goals, signals, metrics	A simple process for choosing product metrics: name the goals, find the signals that show success or failure, and turn the signals into metrics (Rodden et al., 2010).	R05Evaluative researchScores with observations · slide 47 (opens in a new tab)	Rodden, K., Hutchinson, H., & Fu, X. (2010). Measuring the user experience on a large scale: User-centered metrics for web applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2395–2398). ACM. doi
Goodhart’s law	The observation that a measure used as a target stops being a good measure, in Strathern’s (1997) wording. R18 When a measure becomes a target, people optimise the measure: logins rise and learning doesn't. Also known as Campbell's law, after the same point made about social indicators (Campbell, 1979).	R15Learning analyticsActivity is not learning · slide 21 (opens in a new tab) R18Basic statistics for educational technologistsWhere numbers come from · slide 53 (opens in a new tab)	Strathern, M. (1997). ‘Improving ratings’: Audit in the British university system. European Review, 5(3), 305–321. cambridge.org Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. doi
Grammar of schooling	The standard structures of school, such as age-graded classes, separate subjects and fixed lessons, which persist because they work for teachers and which absorb or reject reforms.	R19Adoption and change in schoolsFrom the field · not in the guide	Cuban, L. (2001). Oversold and underused: Computers in the classroom. Harvard University Press. doi Tyack, D., & Tobin, W. (1994). The “grammar” of schooling: Why has it been so hard to change? American Educational Research Journal, 31(3), 453–479. doi
Grapevine	An organisation's informal network of conversation, which carries news of a change faster than official channels and with less accuracy.	R20Communicating change to schoolsFrom the field · not in the guide	Davis, K. (1953). Management communication and the grapevine. Harvard Business Review, 31(5), 43–49.
Groan zone	The uncomfortable middle of a group decision, after ideas have been opened up and before they come together, when people struggle to understand one another's views.	R03Design thinking, used honestlyFrom the field · not in the guide	Kaner, S., Lind, L., Toldi, C., Fisk, S., & Berger, D. (2014). Facilitator's guide to participatory decision-making (3rd ed.). Jossey-Bass.
Grounded theory	A research approach that builds a theory from the data by coding, comparing cases constantly and collecting further data until the emerging categories stop changing.	R06Design methods IFrom the field · not in the guide	Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Aldine.
Grounding	Making a model answer from specific supplied material, so its answers can be traced and checked.	R16AI in learning productsGround and check · slide 45 (opens in a new tab)	Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15, Article 17458. doi Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33.
Guardrail	A rule built into a product that limits what the model will do, such as refusing to give a full solution before an attempt.	R16AI in learning productsProtect the struggle · slide 25 (opens in a new tab)	Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), Article e2422633122. doi Department for Education. (2026). Generative AI: Product safety standards (Updated January 19, 2026). GOV.UK. gov.uk
Guardrail metric	A measure that must not get worse during a test, such as teacher workload or results for the lowest attainers.	R13Experimentation and product analyticsThe metric first · slide 19 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
Guerrilla testing	Quick, informal usability testing with whoever is to hand, such as colleagues or passers-by. It trades a careful sample for speed and low cost.	R05Evaluative researchFrom the field · not in the guide	Krug, S. (2010). Rocket surgery made easy: The do-it-yourself guide to finding and fixing usability problems. New Riders.
Guiding coalition	A group with enough authority, expertise and credibility to lead a change and speak for it, the second of Kotter's eight steps.	R20Communicating change to schoolsFrom the field · not in the guide	Kotter, J. P. (1995). Leading change: Why transformation efforts fail. Harvard Business Review, 73(2), 59–67. hbr.org Kotter, J. P. (1996). Leading change. Harvard Business School Press. doi
Gulfs of execution and evaluation	The gap between what a user wants to do and what the system lets them do, and the gap between the system's state and the user's ability to see and interpret it.	R08UX foundations and design systemsFrom the field · not in the guide	Norman, D. (2013). The design of everyday things (Rev. and expanded ed.). Basic Books. basicbooks.com Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985). Direct manipulation interfaces. Human–Computer Interaction, 1(4), 311–338. doi
H
Hawthorne effect	A change in people's behaviour caused by knowing they are being observed or studied. It is also called the observer effect.	R04Discovery researchFrom the field · not in the guide	McCambridge, J., Witton, J., & Elbourne, D. R. (2014). Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects. Journal of Clinical Epidemiology, 67(3), 267–277. doi
HEART	Happiness, engagement, adoption, retention and task success: categories for product metrics (Rodden et al., 2010).	R05Evaluative researchScores with observations · slide 47 (opens in a new tab)	Rodden, K., Hutchinson, H., & Fu, X. (2010). Measuring the user experience on a large scale: User-centered metrics for web applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2395–2398). ACM. doi
Help and documentation heuristic	Nielsen's tenth usability heuristic: it is best if a system needs no additional explanation, but documentation may be necessary to help users complete their tasks (Nielsen, 2024).	R22Support and feedback loopsHelp in place · slide 26 (opens in a new tab)	Nielsen, J. (2024). 10 usability heuristics for user interface design. Nielsen Norman Group. (Original work published 1994) nngroup.com Kendrick, A. (2020). Help and documentation (Usability heuristic #10). Nielsen Norman Group. nngroup.com
Heuristic evaluation	A review in which evaluators check an interface against a set of usability principles (Nielsen & Molich, 1990).	R08UX foundations and design systemsThe next step · slide 26 (opens in a new tab)	Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 249–256). ACM. doi Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 152–158). ACM. doi Nielsen, J. (2024). 10 usability heuristics for user interface design. Nielsen Norman Group. (Original work published 1994) nngroup.com
Hick's law	The finding that choice reaction time rises with the number of alternatives; a reason to make the main action stand out, not a rule to cut every menu down (Hick, 1952).	R08UX foundations and design systemsThe next step · slide 25 (opens in a new tab)	Hick, W. E. (1952). On the rate of gain of information. Quarterly Journal of Experimental Psychology, 4(1), 11–26. doi Hyman, R. (1953). Stimulus information as a determinant of reaction time. Journal of Experimental Psychology, 45(3), 188–196. doi Liu, W., Gori, J., Rioul, O., Beaudouin-Lafon, M., & Guiard, Y. (2020). How relevant is Hick's law for HCI? In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–11). ACM. doi
High-risk AI system	Under the EU AI Act, a system in a listed use, including evaluating learning outcomes, that carries extra obligations (European Union, 2024).	R16AI in learning productsA human accountable · slide 36 (opens in a new tab)	European Union. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L 2024/1689. eur-lex.europa.eu Council of the European Union. (2026, June 29). Artificial intelligence: Council gives final green light to simplify and streamline rules [Press release]. consilium.europa.eu
Holdout group	A small share of users kept on the old experience for a long period, often across many launches, to measure the long-term or cumulative effect of changes.	R13Experimentation and product analyticsFrom the field · not in the guide	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
Horizontal and vertical prototypes	A horizontal prototype shows the full breadth of a product's features with nothing working behind them. A vertical prototype makes one narrow feature work in depth.	R07Design methods IIFrom the field · not in the guide	Nielsen, J. (1993). Usability engineering. Academic Press. doi
How might we	A question format that turns an insight into an opportunity without naming the solution (IDEO.org, 2015).	R06Design methods IQuestions worth designing for · slide 45 (opens in a new tab)	IDEO.org. (2015). The field guide to human-centered design. IDEO.org. designkit.org Basadur, B. (2021). The origin of "How might we?" Basadur Applied Innovation. basadur.com Rosala, M. (2021). Using "How might we" questions to ideate on the right problems. Nielsen Norman Group. nngroup.com
Human in the loop	A design in which a person can see, question and override an AI system's output (U.S. Department of Education, 2023).	R16AI in learning productsA human accountable · slide 33 (opens in a new tab)	U.S. Department of Education, Office of Educational Technology. (2023). Artificial intelligence and the future of teaching and learning: Insights and recommendations. files.eric.ed.gov Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2024). Tutor CoPilot: A human-AI approach for scaling real-time expertise [Preprint]. arXiv. doi
Human-centred design	An approach to designing systems that starts from the people who will use them, involves them throughout, and revises the design in response to evaluation with them.	R03Design thinking, used honestlyFrom the field · not in the guide	Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92. hbr.org International Organization for Standardization. (2019). Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems (ISO 9241-210:2019).
Human-centred learning analytics	An approach that designs analytics tools with the teachers and students who will use them, so that the tools fit their practice and needs.	R15Learning analyticsFrom the field · not in the guide	Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. doi
Hype cycle	Gartner's picture of how expectations of a new technology rise to a peak, fall into disillusionment and then settle at a realistic level.	R19Adoption and change in schoolsFrom the field · not in the guide	Fenn, J., & Raskino, M. (2008). Mastering the hype cycle: How to choose the right innovation at the right time. Harvard Business Press.
Hypercare	A fixed period of extra support straight after a launch or migration, with more staff, faster targets and daily review of issues.	R22Support and feedback loopsFrom the field · not in the guide
Hypothesis statement	An assumption rewritten in testable form: we believe this outcome will follow if these users get this feature, and we will know we are right when we see this signal.	R10Lean product developmentFrom the field · not in the guide	Gothelf, J., & Seiden, J. (2013). Lean UX: Applying lean principles to improve user experience. O'Reilly Media. oreilly.com Bland, D. J., & Osterwalder, A. (2019). Testing business ideas: A field guide for rapid experimentation. Wiley. books.google.com
I
Ideation	The stage of design work given to producing many possible solutions to a framed problem, before any is judged or chosen.	R03Design thinking, used honestlyFrom the field · not in the guide	Brown, T. (2008). Design thinking. Harvard Business Review, 86(6), 84–92. hbr.org
Identifier	A value that distinguishes one record, such as a student or a school, from every other.	R14Data strategy and governanceIdentifiers first · slide 38 (opens in a new tab)	Department for Education. (2019). Unique pupil number (UPN) (Guide version 1.2). assets.publishing.service.gov.uk Government Accountability Office. (2014). Education and workforce data: Challenges in matching student and worker information raise concerns about longitudinal data systems (GAO-15-27). gao.gov
Ill-structured problem	A problem whose goals, constraints and permitted moves are not fully given at the outset, so that part of solving it is deciding what the problem is.	R03Design thinking, used honestlyFrom the field · not in the guide	Simon, H. A. (1973). The structure of ill structured problems. Artificial Intelligence, 4(3–4), 181–201. doi
Impact	Long-term change in organisations, communities or systems, often years after the work.	R01Digital product managementThe spine · slide 13 (opens in a new tab)	W. K. Kellogg Foundation. (2004). Logic model development guide. naccho.org
Implementation dip	A fall in performance and confidence while new skills are learned (Fullan, 2001).	R19Adoption and change in schoolsExpect variation · slide 40 (opens in a new tab)	Fullan, M. (2001). Leading in a culture of change. Jossey-Bass. Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk
Implementation phases	The EEF's four phases of implementation, Explore, Prepare, Deliver and Sustain, which end in an explicit decision to sustain, scale or de-implement an approach (Sharples et al., 2024).	R19Adoption and change in schoolsPlan the exit · slide 46 (opens in a new tab)	Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk
Improvement science	An approach to improving school systems through rapid cycles of Plan, Do, Study, Act, to learn fast, fail fast and improve quickly (Bryk et al., 2015).	R10Lean product developmentCount learning · slide 40 (opens in a new tab)	Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America's schools can get better at getting better. Harvard Education Press. books.google.com Carnegie Foundation for the Advancement of Teaching. (n.d.). The six core principles of improvement. Retrieved September 17, 2026, from carnegiefoundation.org
In vivo code	A code that takes a participant's own word or phrase as its label, keeping their language in the analysis.	R06Design methods IFrom the field · not in the guide	Saldaña, J. (2025). The coding manual for qualitative researchers (5th ed.). SAGE.
In-app notification	A message shown inside a product while it is in use, such as a banner, badge or pop-up, rather than sent by email or text.	R20Communicating change to schoolsFrom the field · not in the guide
In-context help	Help shown inside the task, at the point where a user needs it, rather than in a separate help centre.	R22Support and feedback loopsHelp in place · slide 28 (opens in a new tab)	Laubheimer, P. (2023). Onboarding tutorials vs. contextual help. Nielsen Norman Group. nngroup.com Kendrick, A. (2020). Help and documentation (Usability heuristic #10). Nielsen Norman Group. nngroup.com Nielsen, J. (2024). 10 usability heuristics for user interface design. Nielsen Norman Group. (Original work published 1994) nngroup.com
In-service training (INSET)	Training for teachers already in post. In England, an INSET day is one of the five days a year when schools close to pupils for staff training.	R21Professional learning that changes practiceFrom the field · not in the guide
Incentive	A payment, voucher or other thanks given to participants for their time. Its size and form can change who volunteers, and in schools it may be restricted.	R04Discovery researchFrom the field · not in the guide	Government Digital Service. (2020). Find user research participants. GOV.UK Service Manual. gov.uk Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com
Incident	An unplanned interruption to a service or reduction in its quality (AXELOS, 2019).	R22Support and feedback loopsThe spine · slide 14 (opens in a new tab)	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com Flora, E. (2023a). An overview of the incident management practice in ITIL 4. Beyond20. beyond20.com
Incident response plan	The agreed steps for a security incident: who decides, how to contain it, how to recover, whom to tell and how to learn from it afterwards.	R17Privacy, security and ethicsFrom the field · not in the guide	Cichonski, P., Millar, T., Grance, T., & Scarfone, K. (2012). Computer security incident handling guide (NIST Special Publication 800-61, Rev. 2). National Institute of Standards and Technology. doi
Inclusive design	Designing so that the widest range of people can use a product, by considering from the start those most likely to be excluded by disability, language, age or circumstance.	R08UX foundations and design systemsFrom the field · not in the guide	Holmes, K. (2018). Mismatch: How inclusion shapes design. MIT Press. doi Horton, S., & Quesenbery, W. (2013). A web for everyone: Designing accessible user experiences. Rosenfeld Media.
Increment	A usable step towards the Product Goal. A sprint may produce several.	R09Agile and Scrum in practiceThe spine · slide 16 (opens in a new tab)	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Induction	The structured programme of support, training and reduced load given to teachers in their first years in the profession or in a new school.	R21Professional learning that changes practiceFrom the field · not in the guide	Ingersoll, R. M., & Strong, M. (2011). The impact of induction and mentoring programs for beginning teachers: A critical review of the research. Review of Educational Research, 81(2), 201–233. doi
Inductive and deductive coding	Inductive coding builds codes from what is in the data. Deductive coding applies codes set in advance from a theory, a framework or the research questions.	R06Design methods IFrom the field · not in the guide	Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. doi Saldaña, J. (2025). The coding manual for qualitative researchers (5th ed.). SAGE.
Inference	Running a trained model to produce an output, as distinct from training it. Every response a student receives is one inference, with its own cost and delay.	R16AI in learning productsFrom the field · not in the guide
Information architecture	The way a product's content is organised, labelled and linked so that people can find what they need and understand where they are.	R08UX foundations and design systemsFrom the field · not in the guide	Rosenfeld, L., Morville, P., & Arango, J. (2015). Information architecture: For the web and beyond (4th ed.). O'Reilly Media.
Information overload	The point at which a person receives more information than they can process, so that decisions get worse and messages are missed.	R20Communicating change to schoolsFrom the field · not in the guide	Eppler, M. J., & Mengis, J. (2004). The concept of information overload: A review of literature from organization science, accounting, marketing, MIS, and related disciplines. The Information Society, 20(5), 325–344. doi
Information radiator	A large, visible display, such as a task board or burndown chart on a wall, that lets anyone see a team's progress without asking.	R09Agile and Scrum in practiceFrom the field · not in the guide	Cockburn, A. (2002). Agile software development. Addison-Wesley. doi
Information scent	The cues in a link, label or heading that tell a user how likely it is to lead to what they are looking for.	R08UX foundations and design systemsFrom the field · not in the guide	Pirolli, P., & Card, S. (1999). Information foraging. Psychological Review, 106(4), 643–675. doi
Infrastructure as code	Defining servers, networks and their settings in files kept under version control, so that environments are created and changed by running tested scripts, not by hand.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com Morris, K. (2016). Infrastructure as code: Managing servers in the cloud. O'Reilly Media.
Innovation accounting	A way of setting milestones and judging progress by what has been learned (Ries, 2011).	R10Lean product developmentCount learning · slide 38 (opens in a new tab)	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com
Innovation configuration	A map, from the Concerns-Based Adoption Model, that describes what an innovation looks like in use, from ideal to unacceptable variations of each component.	R19Adoption and change in schoolsFrom the field · not in the guide	Hall, G. E., & Hord, S. M. (2014). Implementing change: Patterns, principles, and potholes (4th ed.). Pearson. pearson.com
Innovation partnership	A public procurement route that runs in phases with intermediate targets and lets the buyer end the partnership after any phase (Directive 2014/24/EU, 2014).	R10Lean product developmentCount learning · slide 41 (opens in a new tab)	Directive 2014/24/EU. (2014). Directive 2014/24/EU of the European Parliament and of the Council of 26 February 2014 on public procurement and repealing Directive 2004/18/EC. Official Journal of the European Union, L 94, 65–242. eur-lex.europa.eu
Input metric	A measure a team can move directly through its own work and which is believed to drive the north star metric.	R01Digital product managementFrom the field · not in the guide	Amplitude. (2019). The North Star playbook. amplitude.com
Insight	An explanation of why a pattern exists. A claim, not a fact.	R06Design methods IThe spine · slide 13 (opens in a new tab)	Kolko, J. (2010). Abductive thinking and sensemaking: The drivers of design synthesis. Design Issues, 26(1), 15–28. doi Pidcock, D. (2018). What is atomic UX research? Prototypr. blog.prototypr.io
Institutional review board (IRB)	A committee that reviews planned research with human participants to protect their rights and welfare. Called a research ethics committee in the UK.	R13Experimentation and product analyticsFrom the field · not in the guide	Protection of Human Subjects, 45 C.F.R. pt. 46 (2018). ecfr.gov British Educational Research Association. (2024). Ethical guidelines for educational research (5th ed.). bera.ac.uk
Institutionalisation	The stage at which a change has become part of an organisation's routines, budgets and expectations and no longer depends on the people who introduced it.	R19Adoption and change in schoolsFrom the field · not in the guide	Fullan, M. (2007). The new meaning of educational change (4th ed.). Teachers College Press.
Instructional rounds	A practice adapted from medical rounds, in which a group of educators visits classrooms to observe a shared problem of practice and then discusses the patterns seen.	R21Professional learning that changes practiceFrom the field · not in the guide	City, E. A., Elmore, R. F., Fiarman, S. E., & Teitel, L. (2009). Instructional rounds in education: A network approach to improving teaching and learning. Harvard Education Press.
Instrumentation	The code that records user and system events in a product, and the work of adding it. An experiment can only measure what has been instrumented.	R13Experimentation and product analyticsFrom the field · not in the guide	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
Integration test	An automated test that checks several parts of a system working together, such as the code and its database, rather than one unit on its own.	R11Software development for non-engineersFrom the field · not in the guide	Fowler, M. (2012, May 1). Test pyramid. martinfowler.com. martinfowler.com Myers, G. J. (1979). The art of software testing. Wiley.
Intelligent tutoring system	Software that guides a learner step by step and adapts to their responses. It predates generative AI (VanLehn, 2011).	R16AI in learning productsDate the evidence · slide 40 (opens in a new tab)	VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. doi Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167–207. doi
Intention to treat	Analysing everyone by the group they were assigned to, whatever they actually used (Education Endowment Foundation, 2022).	R13Experimentation and product analyticsThe right unit · slide 28 (opens in a new tab)	Education Endowment Foundation. (2022). Statistical analysis guidance for EEF evaluations. d2tic4wvo1iusb.cloudfront.net What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0 (WWC 2022008). U.S. Department of Education, Institute of Education Sciences. ies.ed.gov
Intercept survey	A short survey shown to people while they are using a website or product, so that answers are gathered in the moment of use.	R05Evaluative researchFrom the field · not in the guide	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com
Internal validity	The extent to which a study supports the claim that the intervention, and not something else, caused the observed difference.	R13Experimentation and product analyticsFrom the field · not in the guide	Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Interoperability standard	An agreed format and method for exchanging a kind of data between systems, such as OneRoster or LTI.	R14Data strategy and governanceIdentifiers first · slide 39 (opens in a new tab)	1EdTech Consortium. (n.d.-b). OneRoster. Retrieved September 17, 2026, from 1edtech.org 1EdTech Consortium. (n.d.-c). Learning Tools Interoperability. Retrieved September 17, 2026, from 1edtech.org IEEE. (2023). IEEE standard for learning technology: JavaScript Object Notation (JSON) data model format and Representational State Transfer (RESTful) web service for learner experience data tracking and access (IEEE 9274.1.1-2023). standards.ieee.org
Interpretation	A reading of what an observation means. It should be recorded separately and tested.	R04Discovery researchNotes and meanings · slide 36 (opens in a new tab)	Ross, R. (1994). The ladder of inference. In P. M. Senge, A. Kleiner, C. Roberts, R. B. Ross, & B. J. Smith, The fifth discipline fieldbook: Strategies and tools for building a learning organization. Currency Doubleday. archive.org Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220. doi
Interquartile range (IQR)	The distance between the 25th and 75th percentiles, which holds the middle half of the values. A measure of spread that outliers do not distort.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley. doi
Interrupted time series	A quasi-experimental design that takes many measurements before and after an intervention and looks for a change in level or trend at the point it was introduced.	R13Experimentation and product analyticsFrom the field · not in the guide	Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Interval	The range within which an estimate plausibly lies. A narrow interval means more certainty.	R15Learning analyticsShow the uncertainty · slide 32 (opens in a new tab)	Perdomo, J. C., Britton, T., Hardt, M., & Abebe, R. (2025). Difficult lessons on social prediction from Wisconsin public schools. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (pp. 2682–2704). ACM. doi Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge. doi
Intervention	The action taken after analytics identifies a need, such as a tutor's call, a message or extra teaching. Analytics without one changes nothing for learners.	R15Learning analyticsFrom the field · not in the guide	Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 134–138). ACM. doi Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267–270). ACM. doi
Intraclass correlation (ICC)	The share of the variation in an outcome that lies between clusters rather than within them.	R13Experimentation and product analyticsSize for clusters · slide 37 (opens in a new tab)	Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. doi Donner, A., & Klar, N. (2000). Design and analysis of cluster randomization trials in health research. Arnold. wiley.com
Intrinsic load	Load caused by the complexity of the material itself.	R08UX foundations and design systemsThe spine · slide 13 (opens in a new tab)	Sweller, J., van Merriënboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive architecture and instructional design. Educational Psychology Review, 10(3), 251–296. doi Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31(2), 261–292. doi
Inverted pyramid	A structure that starts with the most important point and tapers to detail.	R20Communicating change to schoolsLead with the teacher · slide 20 (opens in a new tab)	Government Digital Service. (n.d.-c). Create a clear structure for your content. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk Nielsen, J. (1997). How users read on the web. Nielsen Norman Group. nngroup.com
INVEST	A test for user stories: independent, negotiable, valuable, estimable, small and testable (Wake, 2003).	R09Agile and Scrum in practiceShip small · slide 20 (opens in a new tab)	Wake, B. (2003, August 17). INVEST in good stories, and SMART tasks. XP123. xp123.com Cohn, M. (2004). User stories applied: For agile software development. Addison-Wesley. mountaingoatsoftware.com
Invisible inventory	Unfinished product development work is physically and financially invisible: it waits in queues, takes no space and appears on no balance sheet (Reinertsen, 2009).	R09Agile and Scrum in practiceMake work visible · slide 27 (opens in a new tab)	Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com
ISO/IEC 27001	The international standard for an information security management system. An organisation is certified against it by an accredited auditor, and buyers often ask vendors for the certificate.	R17Privacy, security and ethicsFrom the field · not in the guide	International Organization for Standardization & International Electrotechnical Commission. (2022). Information security, cybersecurity and privacy protection — Information security management systems — Requirements (ISO/IEC 27001:2022).
Iterative design	Designing in repeated rounds of making, testing with users and revising, with each version built on what the last test showed.	R07Design methods IIFrom the field · not in the guide	Buxton, B. (2007). Sketching user experiences: Getting the design right and the right design. Morgan Kaufmann. books.google.com Nielsen, J. (1993). Iterative user-interface design. Computer, 26(11), 32–41. doi
ITIL	The most widely used framework for IT service management, which separates restoring normal service after an incident from finding and removing the causes of incidents (AXELOS, 2019).	R22Support and feedback loopsThe spine · slide 14 (opens in a new tab)	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com Flora, E. (2023a). An overview of the incident management practice in ITIL 4. Beyond20. beyond20.com Flora, E. (2023b). An overview of the problem management practice in ITIL 4. Beyond20. beyond20.com
J
Jakob's law	The observation that users spend most of their time on other products, so they expect a new one to work the way those already do.	R08UX foundations and design systemsFrom the field · not in the guide	Yablonski, J. (2020). Laws of UX: Using psychology to design better products & services. O'Reilly Media.
Job to be done	The progress a person is trying to make in a particular circumstance (Christensen et al., 2016).	R04Discovery researchThe spine · slide 13 (opens in a new tab) R06Design methods IThe job, not the profile · slide 39 (opens in a new tab)	Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Competing against luck: The story of innovation and customer choice. HarperBusiness. archive.org Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Know your customers' "jobs to be done". Harvard Business Review, 94(9), 54–62. hbr.org Christensen, C. M., & Raynor, M. E. (2003). The innovator's solution: Creating and sustaining successful growth. Harvard Business School Press.
Job-embedded professional development	Professional learning that takes place during the working day, in the teacher's own school and on their own teaching, rather than at an external course.	R21Professional learning that changes practiceFrom the field · not in the guide	Darling-Hammond, L., Hyler, M. E., & Gardner, M. (2017). Effective teacher professional development. Learning Policy Institute. doi
Journey map	A picture of what one person does, thinks and feels while completing a task. R06 A picture of the process a person goes through to reach a goal (Gibbons, 2018).	R02Service design and systems thinkingThe spine · slide 13 (opens in a new tab) R06Design methods IThe job, not the profile · slide 40 (opens in a new tab)	Kalbach, J. (2020). Mapping experiences: A complete guide to customer alignment through journeys, blueprints, and diagrams (2nd ed.). O'Reilly Media. openlibrary.org Stickdorn, M., Hormess, M. E., Lawrence, A., & Schneider, J. (2018). This is service design doing: Applying service design thinking in the real world. O'Reilly Media. Gibbons, S. (2018). Journey mapping 101. Nielsen Norman Group. nngroup.com
Just-in-time	Producing only what is needed, when it is needed and in the amount needed, so that unfinished work and stock do not pile up. One of the two pillars of the Toyota Production System.	R10Lean product developmentFrom the field · not in the guide	Ohno, T. (1988). Toyota production system: Beyond large-scale production. Productivity Press. Womack, J. P., Jones, D. T., & Roos, D. (1990). The machine that changed the world. Rawson Associates. archive.org
K
k-anonymity	A property of a released dataset in which each record matches at least k−1 others on the attributes that could identify someone, so that no one stands alone.	R17Privacy, security and ethicsFrom the field · not in the guide	Sweeney, L. (2002). k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(5), 557–570. doi
Kaizen	Continuous improvement through many small changes proposed and tested by the people who do the work, rather than through occasional large redesigns.	R10Lean product developmentFrom the field · not in the guide	Imai, M. (1986). Kaizen: The key to Japan's competitive success. Random House. Liker, J. K. (2004). The Toyota way: 14 management principles from the world's greatest manufacturer. McGraw-Hill.
Kanban	A way of managing work that makes it visible on a board and limits how much is in progress at once (Anderson, 2010). R10 A method for managing work that makes it visible and limits how much is in progress (Anderson, 2010).	R09Agile and Scrum in practiceMake work visible · slide 26 (opens in a new tab) R10Lean product developmentWork in progress · slide 32 (opens in a new tab)	Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Blue Hole Press. books.google.com Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com Ohno, T. (1988). Toyota production system: Beyond large-scale production. Productivity Press.
Kano model	A view of satisfaction as two-dimensional: must-be qualities anger users when absent and add little when present, while attractive qualities delight when present and are not missed when absent (Kano et al., 1984).	R01Digital product managementSay no in public · slide 32 (opens in a new tab)	Kano, N., Seraku, N., Takahashi, F., & Tsuji, S. (1984). Attractive quality and must-be quality [in Japanese]. Journal of the Japanese Society for Quality Control, 14(2), 147–156. doi
Key message	One of the few points a communicator wants every reader to remember, agreed in advance and repeated in the same words across channels.	R20Communicating change to schoolsFrom the field · not in the guide
Key performance indicator (KPI)	A measure an organisation tracks routinely to judge whether an activity is performing as intended, usually against a set target.	R01Digital product managementFrom the field · not in the guide	Parmenter, D. (2020). Key performance indicators: Developing, implementing, and using winning KPIs (4th ed.). Wiley. doi
Kirkpatrick model	A model that evaluates training at four levels: participants' reaction, their learning, their behaviour on the job, and results for the organisation.	R21Professional learning that changes practiceFrom the field · not in the guide	Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45–51. ascd.org Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. Berrett-Koehler.
KJ method	The affinity method developed by the ethnologist Jiro Kawakita for making sense of field data (Scupin, 1997).	R06Design methods ISynthesise together · slide 19 (opens in a new tab)	Scupin, R. (1997). The KJ method: A technique for analyzing data derived from Japanese ethnology. Human Organization, 56(2), 233–237. doi
Knowledge base	A searchable collection of articles on how to use a product and how to fix known problems, used by support staff, by users, or by both.	R22Support and feedback loopsFrom the field · not in the guide	Consortium for Service Innovation. (2016). KCS v6 practices guide. library.serviceinnovation.org
Knowledge tracing	Estimating, from a student's sequence of right and wrong answers, the probability that they have mastered each skill. Adaptive systems use it to choose the next task.	R16AI in learning productsFrom the field · not in the guide	Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278. doi
Knowledge-Centered Service	A support method with two loops: the Solve Loop, the request-response workflow, and the Evolve Loop, which analyses patterns across many requests to improve products, processes and policies (Consortium for Service Innovation, 2016).	R22Support and feedback loopsTickets as research · slide 21 (opens in a new tab)	Consortium for Service Innovation. (2016). KCS v6 practices guide. library.serviceinnovation.org
Known error	A problem whose cause has been analysed but not yet removed, usually with a workaround in place (AXELOS, 2019).	R22Support and feedback loopsThe spine · slide 14 (opens in a new tab)	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com Flora, E. (2023b). An overview of the problem management practice in ITIL 4. Beyond20. beyond20.com
L
Lab and field testing	The choice between testing in a lab and testing where the product is used; the evidence is mixed, and the setting has to be chosen on purpose (Kjeldskov & Skov, 2014).	R05Evaluative researchThe real pace, the real room · slide 39 (opens in a new tab)	Kjeldskov, J., Skov, M. B., Als, B. S., & Høegh, R. T. (2004). Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field. In S. Brewster & M. Dunlop (Eds.), Mobile human-computer interaction: MobileHCI 2004 (pp. 61–73). Springer. doi Kjeldskov, J., & Skov, M. B. (2014). Was it worth the hassle? Ten years of mobile HCI research discussions on lab and field evaluations. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 43–52). ACM. doi
Ladder of inference	The steps from observable data to beliefs and actions, each adding the observer's meaning (Ross, 1994).	R04Discovery researchNotes and meanings · slide 37 (opens in a new tab)	Ross, R. (1994). The ladder of inference. In P. M. Senge, A. Kleiner, C. Roberts, R. B. Ross, & B. J. Smith, The fifth discipline fieldbook: Strategies and tools for building a learning organization. Currency Doubleday. archive.org Argyris, C. (1990). Overcoming organizational defenses: Facilitating organizational learning. Allyn and Bacon.
Laddering	An interview technique that repeatedly asks why something matters, moving from a feature a person mentions to the consequence and then the value behind it.	R04Discovery researchFrom the field · not in the guide	Reynolds, T. J., & Gutman, J. (1988). Laddering theory, method, analysis, and interpretation. Journal of Advertising Research, 28(1), 11–31. doi
Lag	The delay between a signal and the outcome it is meant to predict or improve.	R15Learning analyticsThe spine · slide 16 (opens in a new tab)	Perdomo, J. C., Britton, T., Hardt, M., & Abebe, R. (2025). Difficult lessons on social prediction from Wisconsin public schools. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (pp. 2682–2704). ACM. doi
Large language model (LLM)	A neural network trained on very large amounts of text to predict the next token, which lets it generate, summarise and transform language on request.	R16AI in learning productsFrom the field · not in the guide	Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. doi Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. doi
Large-Scale Scrum (LeSS)	Scrum applied to many teams working together on one product, with fewer roles and artefacts, not more (Larman & Vodde, n.d.).	R09Agile and Scrum in practiceInspect the process · slide 34 (opens in a new tab)	Larman, C., & Vodde, B. (n.d.). Introduction to LeSS. LeSS Company. Retrieved September 17, 2026, from less.works Schwaber, K. (2013, August 6). unSAFe at any speed. Ken Schwaber's Blog. kenschwaber.wordpress.com
Last responsible moment	The point beyond which delaying a decision would close off an important option. Lean teams defer irreversible decisions until then, so they are made with the most information.	R10Lean product developmentFrom the field · not in the guide	Poppendieck, M., & Poppendieck, T. (2003). Lean software development: An agile toolkit. Addison-Wesley. informit.com
Latency	The delay between sending a request and receiving the response. Users feel it as slowness, and it usually climbs steeply as a system nears its capacity.	R11Software development for non-engineersFrom the field · not in the guide	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google Kleppmann, M. (2017). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. O'Reilly Media.
Lawful basis	The legal ground an organisation relies on to process personal data, such as consent, contract, legal obligation, public task or legitimate interests. Schools seldom need to rely on consent.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88.
Lead user	A user whose needs are ahead of most others' and who has often already improvised a solution, making them a source of ideas that the wider group will later want.	R03Design thinking, used honestlyFrom the field · not in the guide	von Hippel, E. (1986). Lead users: A source of novel product concepts. Management Science, 32(7), 791–805. doi
Lead-in time	The time between announcing a change and the date it takes effect.	R20Communicating change to schoolsTiming · slide 33 (opens in a new tab)	Department for Education. (2017). Department for Education protocol for changes to accountability, curriculum and qualifications (DFE-00114-2015). gov.uk Green, M. (2025). DfE should stop publishing statutory guidance for schools in July. Tes. tes.com
Leading question	A question whose wording suggests the answer the asker hopes for.	R05Evaluative researchQuestions that cannot lead · slide 31 (opens in a new tab)	Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press. archive.org Pew Research Center. (2021). Writing survey questions. pewresearch.org
Leading signal	A behaviour that moves within a planning cycle and plausibly moves before the result you want.	R01Digital product managementThe leading signal · slide 43 (opens in a new tab)	Amplitude. (2019). The North Star playbook. amplitude.com W. K. Kellogg Foundation. (2004). Logic model development guide. naccho.org
Leaky funnel	This guide's picture of adoption: people lost between awareness, trial, use and changed practice.	R19Adoption and change in schoolsThe spine · slide 14 (opens in a new tab)	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press. Cuban, L. (2001). Oversold and underused: Computers in the classroom. Harvard University Press. doi
Lean canvas	An adaptation of the business model canvas for new products, which replaces some blocks with problem, solution, key metrics and unfair advantage.	R10Lean product developmentFrom the field · not in the guide	Maurya, A. (2012). Running lean: Iterate from plan A to a plan that works (2nd ed.). O'Reilly Media.
Lean start-up	An approach built on three practices: writing the business idea as hypotheses, testing them with customers outside the building, and building in small steps (Blank, 2013).	R10Lean product developmentThe misconception · slide 10 (opens in a new tab)	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com Blank, S. (2013). Why the lean start-up changes everything. Harvard Business Review, 91(5), 63–72. hbr.org Camuffo, A., Cordova, A., Gambardella, A., & Spina, C. (2020). A scientific approach to entrepreneurial decision making: Evidence from a randomized control trial. Management Science, 66(2), 564–586. doi
Lean thinking	Lean production reduced to five principles: value, the value stream, flow, pull and perfection (Womack & Jones, 1996).	R10Lean product developmentThe spine · slide 15 (opens in a new tab)	Womack, J. P., & Jones, D. T. (1996). Lean thinking: Banish waste and create wealth in your corporation. Simon & Schuster. lean.org Womack, J. P., Jones, D. T., & Roos, D. (1990). The machine that changed the world. Rawson Associates. archive.org
Learnability	How quickly and easily a new user can reach competent use of a product. It is measured by how performance improves over repeated attempts.	R05Evaluative researchFrom the field · not in the guide	Nielsen, J. (1993). Usability engineering. Academic Press. doi
Learner model	A system's running estimate of what a student knows, can do or prefers, updated from their responses and used to adapt tasks or report progress.	R15Learning analyticsFrom the field · not in the guide	Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278. doi
Learning analytics	The measurement, collection, analysis and reporting of data about learners and their contexts, to understand and improve learning (Long and Siemens, 2011).	R15Learning analyticsThe misconception · slide 8 (opens in a new tab)	Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5). er.educause.edu Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. doi Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317. doi
Learning analytics cycle	Learners generate data, the data becomes metrics or visualisations, and those lead to interventions that affect learners. A dashboard nobody acts on stops the cycle halfway (Clow, 2012).	R15Learning analyticsThe action first · slide 26 (opens in a new tab)	Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 134–138). ACM. doi
Learning design	The planned sequence of tasks, resources and support in a course. Analytics can only be interpreted against what the design intended students to do.	R15Learning analyticsFrom the field · not in the guide	Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68–84. doi Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. doi
Learning evidence	Information designed to show what a student understands or can do, such as answers to well-designed questions.	R15Learning analyticsThe misconception · slide 10 (opens in a new tab)	Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. doi Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68–84. doi
Learning management system (LMS)	The platform through which an institution delivers courses, materials, assignments and grades, and the main source of activity data. Called a virtual learning environment (VLE) in the UK.	R15Learning analyticsFrom the field · not in the guide	Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. doi
Learning record store (LRS)	A database that receives, stores and returns records of learning activity sent as xAPI statements from many tools.	R15Learning analyticsFrom the field · not in the guide	IEEE. (2023). IEEE standard for learning technology: JavaScript Object Notation (JSON) data model format and Representational State Transfer (RESTful) web service for learner experience data tracking and access (IEEE 9274.1.1-2023).
Learning Tools Interoperability (LTI)	A standard that lets learning management systems or platforms integrate remote tools and content in a standard way, such as opening a tool from the LMS (1EdTech Consortium, n.d.-c).	R14Data strategy and governanceIdentifiers first · slide 39 (opens in a new tab)	1EdTech Consortium. (n.d.-c). Learning Tools Interoperability. Retrieved September 17, 2026, from 1edtech.org
Learning walk	A short, structured visit to several classrooms by leaders or colleagues, looking for one agreed aspect of teaching rather than judging individual teachers.	R21Professional learning that changes practiceFrom the field · not in the guide
Least privilege	The principle that each user, program or service gets only the access its job needs, so that a mistake or a stolen account can do limited damage.	R17Privacy, security and ethicsFrom the field · not in the guide	Saltzer, J. H., & Schroeder, M. D. (1975). The protection of information in computer systems. Proceedings of the IEEE, 63(9), 1278–1308. doi
Legacy system	An older system that an organisation still depends on but finds hard to change, because its technology is outdated, its authors have left or it lacks tests and documentation.	R11Software development for non-engineersFrom the field · not in the guide	Feathers, M. C. (2004). Working effectively with legacy code. Prentice Hall. doi Bisbal, J., Lawless, D., Wu, B., & Grimson, J. (1999). Legacy information systems: Issues and directions. IEEE Software, 16(5), 103–111. doi
Lesson study	A Japanese form of professional learning in which teachers jointly plan a lesson, watch one of them teach it, and discuss what the students did.	R21Professional learning that changes practiceFrom the field · not in the guide	Lewis, C., Perry, R., & Murata, A. (2006). How should research contribute to instructional improvement? The case of lesson study. Educational Researcher, 35(3), 3–14. doi Stigler, J. W., & Hiebert, J. (1999). The teaching gap: Best ideas from the world's teachers for improving education in the classroom. Free Press.
Levels of use	Eight levels describing how a person uses an innovation, from nonuse to renewal (Hall & Hord, 2014).	R19Adoption and change in schoolsExpect variation · slide 39 (opens in a new tab)	Hall, G. E., & Hord, S. M. (2014). Implementing change: Patterns, principles, and potholes (4th ed.). Pearson. pearson.com Hall, G. E., Loucks, S. F., Rutherford, W. L., & Newlove, B. W. (1975). Levels of use of the innovation: A framework for analyzing innovation adoption. Journal of Teacher Education, 26(1), 52–56. doi
Leverage point	A place in a system where a small change can produce a large effect (Meadows, 1999).	R02Service design and systems thinkingPolicy as material · slide 36 (opens in a new tab)	Meadows, D. (1999). Leverage points: Places to intervene in a system. The Sustainability Institute. donellameadows.org Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green.
Likert scale	A set of statements, each rated on an ordered scale from strongly disagree to strongly agree, with the ratings combined into one score. The name is often used loosely for a single such item.	R05Evaluative researchFrom the field · not in the guide	Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 1–55.
Line of interaction	The line on a service blueprint that separates what the user does from what frontstage staff or systems do. Every crossing of it is a touchpoint.	R02Service design and systems thinkingFrom the field · not in the guide	Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008). Service blueprinting: A practical technique for service innovation. California Management Review, 50(3), 66–94. doi Gibbons, S. (2017, August 27). Service blueprints: Definition. Nielsen Norman Group. nngroup.com
Line of internal interaction	The line on a service blueprint that separates backstage staff, who deal with the user's case, from the support processes and systems they rely on.	R02Service design and systems thinkingFrom the field · not in the guide	Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008). Service blueprinting: A practical technique for service innovation. California Management Review, 50(3), 66–94. doi Gibbons, S. (2017, August 27). Service blueprints: Definition. Nielsen Norman Group. nngroup.com
Line of visibility	The line on a blueprint that separates what the user sees from what they don't.	R02Service design and systems thinkingMap the backstage · slide 19 (opens in a new tab)	Shostack, G. L. (1984). Designing services that deliver. Harvard Business Review, 62(1), 133–139. hbr.org Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008). Service blueprinting: A practical technique for service innovation. California Management Review, 50(3), 66–94. doi
Lineage	A record of where data came from and what has been done to it on the way.	R14Data strategy and governanceCollect less · slide 31 (opens in a new tab)	Government Data Quality Hub. (2020). The Government Data Quality Framework. GOV.UK. gov.uk DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Linear regression	A method that fits a straight-line relationship between an outcome and one or more predictors, estimating how much the outcome changes as each predictor changes.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer. Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press. doi
Linked session	A training session built around a lesson the teacher is about to teach.	R21Professional learning that changes practiceThe teacher's own lesson · slide 20 (opens in a new tab)	Desimone, L. M., & Garet, M. S. (2015). Best practices in teachers' professional development in the United States. Psychology, Society, & Education, 7(3), 252–263. repositorio.ual.es Desimone, L. M. (2009). Improving impact studies of teachers' professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199. doi
Little's law	In a stable system, average work in progress equals the arrival rate multiplied by the average time in the system (Little, 1961).	R10Lean product developmentWork in progress · slide 32 (opens in a new tab)	Little, J. D. C. (1961). A proof for the queuing formula: L = λW. Operations Research, 9(3), 383–387. doi Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com
Load balancer	A component that spreads incoming requests across several servers, so that no single one is overwhelmed and a failed server can be taken out of use.	R11Software development for non-engineersFrom the field · not in the guide	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Load test	A test that sends a system realistic or higher-than-expected traffic to see how it copes and where it breaks.	R11Software development for non-engineersThe busiest day · slide 31 (opens in a new tab)	Government Digital Service. (2017, February 6). Test your service's performance. GOV.UK Service Manual. gov.uk Office of Inspector General. (2016). HealthCare.gov: Case study of CMS management of the federal marketplace (OEI-06-14-00350). U.S. Department of Health and Human Services. oig.hhs.gov U.S. Government Accountability Office. (2014). Healthcare.gov: Ineffective planning and oversight practices underscore the need for improved contract management (GAO-14-694). gao.gov
Logic model	A chain that separates outputs, the direct products of activities, from outcomes, the changes in participants' behaviour, and impact, the change in organisations, communities or systems (W. K. Kellogg Foundation, 2004).	R01Digital product managementThe spine · slide 13 (opens in a new tab)	W. K. Kellogg Foundation. (2004). Logic model development guide. naccho.org
Longitudinal data system	A system that links records about the same learners across years, institutions and sometimes sectors, so that their progress can be followed over time.	R14Data strategy and governanceFrom the field · not in the guide	Government Accountability Office. (2014). Education and workforce data: Challenges in matching student and worker information raise concerns about longitudinal data systems (GAO-15-27). gao.gov
Loose coupling	The weak connection between what a school system decides centrally and what happens in each classroom, which lets teachers buffer their practice from outside change.	R19Adoption and change in schoolsFrom the field · not in the guide	Weick, K. E. (1976). Educational organizations as loosely coupled systems. Administrative Science Quarterly, 21(1), 1–19. doi
Lundy model of participation	Four conditions for children's participation to mean something: space to form and express a view, support to express it, an audience with the power to act, and influence (Lundy, 2007).	R04Discovery researchTwo yeses · slide 45 (opens in a new tab)	Lundy, L. (2007). ‘Voice’ is not enough: Conceptualising Article 12 of the United Nations Convention on the Rights of the Child. British Educational Research Journal, 33(6), 927–942. doi United Nations. (1989). Convention on the Rights of the Child (General Assembly resolution 44/25). ohchr.org
M
Machine learning	Building systems that learn patterns from examples instead of following rules written by hand. Most current AI, including generative AI, is machine learning.	R16AI in learning productsFrom the field · not in the guide	Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255–260. doi
Maintenance window	A period agreed and announced in advance during which a service may be changed or taken offline, chosen for when the fewest people need it.	R12DevOps and continuous deliveryFrom the field · not in the guide	Limoncelli, T. A., & Hogan, C. (2001). The practice of system and network administration. Addison-Wesley.
Major incident	An incident with serious impact on many users, handled by a separate procedure with a named lead, a dedicated team and frequent updates.	R22Support and feedback loopsFrom the field · not in the guide	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com
Management information system (MIS)	The core administrative system a school uses to hold pupil, staff, attendance and assessment records. Called a student information system (SIS) in North America.	R14Data strategy and governanceFrom the field · not in the guide	Department for Education. (2018). Making data work: Report of the Teacher Workload Advisory Group. assets.publishing.service.gov.uk
Mandate	An instruction from an authority to deliver something, often framed as a solution.	R01Digital product managementWork back from the mandate · slide 36 (opens in a new tab)	Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Competing against luck: The story of innovation and customer choice. HarperBusiness. archive.org Morozov, E. (2013). To save everything, click here: The folly of technological solutionism. PublicAffairs.
Margin of error	The amount added to and subtracted from a survey estimate to give its confidence interval, usually at 95 per cent. It shrinks as the sample grows.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. doi
Master data	The shared core records that other data refers to: students, staff, schools, classes and courses.	R14Data strategy and governanceIdentifiers first · slide 37 (opens in a new tab)	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Master–apprentice model	The relationship a researcher adopts in contextual inquiry, learning the work as an apprentice would, while it is under way (Holtzblatt & Beyer, 2013).	R04Discovery researchWatch first · slide 20 (opens in a new tab)	Holtzblatt, K., & Beyer, H. R. (2013). Contextual design. In M. Soegaard & R. F. Dam (Eds.), The encyclopedia of human-computer interaction (2nd ed.). Interaction Design Foundation. ixdf.org Beyer, H., & Holtzblatt, K. (1998). Contextual design: Defining customer-centered systems. Morgan Kaufmann. archive.org
Mean	The sum of a set of values divided by how many there are. It is what most people call the average, and a few extreme values can pull it a long way.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Wheelan, C. (2013). Naked statistics: Stripping the dread from the data. W. W. Norton.
Mechanism	An evidence-based ingredient of how people learn and change behaviour, such as feedback or goal setting.	R21Professional learning that changes practiceThe second week · slide 47 (opens in a new tab)	Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov
Media richness	A channel's capacity to carry cues and allow quick feedback. Face-to-face talk is rich and a bulk email is lean; ambiguous messages need richer channels.	R20Communicating change to schoolsFrom the field · not in the guide	Daft, R. L., & Lengel, R. H. (1986). Organizational information requirements, media richness and structural design. Management Science, 32(5), 554–571. doi
Median	The middle value when a set of values is put in order. Half lie above it and half below, so extreme values barely move it.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Wheelan, C. (2013). Naked statistics: Stripping the dread from the data. W. W. Norton.
Member checking	Taking findings or interpretations back to the people they came from, to check that they recognise the account as accurate.	R06Design methods IFrom the field · not in the guide	Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. SAGE. doi
Mental model	A person's internal picture of how something works, which shapes what they notice and what they do, whether or not it is accurate.	R02Service design and systems thinkingFrom the field · not in the guide	Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. Doubleday/Currency. Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Harvard University Press.
Mental model diagram	A diagram that lays out what people do, think and feel in pursuit of a goal, grouped into towers, with the organisation's features and services lined up beneath.	R06Design methods IFrom the field · not in the guide	Kalbach, J. (2020). Mapping experiences: A complete guide to customer alignment through journeys, blueprints, and diagrams (2nd ed.). O'Reilly Media. openlibrary.org Young, I. (2008). Mental models: Aligning design strategy with human behavior. Rosenfeld Media. doi
Mentoring	Support given to a less experienced teacher by a more experienced one over a sustained period, covering the role as a whole rather than one technique.	R21Professional learning that changes practiceFrom the field · not in the guide	Hobson, A. J., Ashby, P., Malderez, A., & Tomlinson, P. D. (2009). Mentoring beginning teachers: What we know and what we don't. Teaching and Teacher Education, 25(1), 207–216. doi
Meta-analysis	A statistical method that pools the effect sizes from many studies of the same question into one weighted average.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3–8. doi Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley. doi
Metadata	Data that describes data: what a field means, its source, its owner and its quality.	R14Data strategy and governanceCollect less · slide 31 (opens in a new tab)	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Government Data Quality Hub. (2020). The Government Data Quality Framework. GOV.UK. gov.uk
Micro-credential	A record of assessed learning in a small, specific skill, often issued as a digital badge, which may stand alone or count towards a larger qualification.	R21Professional learning that changes practiceFrom the field · not in the guide	Oliver, B. (2022). Towards a common definition of micro-credentials. UNESCO.
Microservices	An architecture that splits a system into small services, each run and deployed separately and talking over a network. The alternative, one application deployed as a whole, is called a monolith.	R11Software development for non-engineersFrom the field · not in the guide	Newman, S. (2015). Building microservices: Designing fine-grained systems. O'Reilly Media.
Microteaching	Teaching a short, scaled-down lesson to a few peers or students, often recorded, and then receiving feedback on one specific skill.	R21Professional learning that changes practiceFrom the field · not in the guide	Allen, D. W., & Ryan, K. (1969). Microteaching. Addison-Wesley. doi
Migration	Moving existing data or systems into a new structure or location.	R11Software development for non-engineersThe cost of change · slide 19 (opens in a new tab)
Miller's law	The much-quoted claim that people can hold about seven items in short-term memory. Later research puts the limit nearer four chunks, and it is not a rule for menu length.	R08UX foundations and design systemsFrom the field · not in the guide	Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81–97. doi Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24(1), 87–114. doi Yablonski, J. (2020). Laws of UX: Using psychology to design better products & services. O'Reilly Media.
Minimum detectable effect	The smallest true effect a test is likely to detect, given its size, design and power.	R13Experimentation and product analyticsSize for clusters · slide 39 (opens in a new tab)	Hedges, L. V., & Hedberg, E. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. doi Bloom, H. S. (1995). Minimum detectable effects: A simple way to report the statistical power of experimental designs. Evaluation Review, 19(5), 547–556. doi
Minimum viable product	The version of a product that gets the most validated learning for the least effort (Ries, 2009).	R10Lean product developmentThe misconception · slide 8 (opens in a new tab)	Ries, E. (2009, August 3). Minimum viable product: A guide. Startup Lessons Learned. startuplessonslearned.com Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com Blank, S. (2013). Why the lean start-up changes everything. Harvard Business Review, 91(5), 63–72. hbr.org
Missed case	A student who went on to fail or leave but was never flagged. Also called a false negative.	R15Learning analyticsThe base rate · slide 41 (opens in a new tab)	Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi
Missing data	Values that were never recorded, such as pupils absent on test day. If they are missing for a reason linked to the outcome, analysing only complete cases biases the result.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. doi Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). Wiley.
Mobile first	Designing for the smallest screen and slowest connection first, then adding to the design for larger screens, which forces early decisions about what matters most.	R08UX foundations and design systemsFrom the field · not in the guide	Wroblewski, L. (2011). Mobile first. A Book Apart.
Mockup	A static picture of a screen showing the intended visual design in full, which looks finished but does nothing.	R07Design methods IIFrom the field · not in the guide	Warfel, T. Z. (2009). Prototyping: A practitioner's guide. Rosenfeld Media.
Model card	A short document published with a model that states its intended uses, its limits and how it performed for different groups of people.	R16AI in learning productsFrom the field · not in the guide	Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, 220–229. doi
Modelling	Showing teachers what a practice looks like, through a live demonstration, a video or a worked example, before asking them to try it.	R21Professional learning that changes practiceFrom the field · not in the guide	Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov
Moderated and unmoderated testing	In a moderated test a researcher guides each participant through the session, in person or remotely. In an unmoderated test participants work through the tasks alone, usually through an online tool.	R05Evaluative researchFrom the field · not in the guide	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Wiley.
Modular contracting	Buying a large system as a series of small contracts, each delivering working parts.	R09Agile and Scrum in practiceContracts that flex · slide 46 (opens in a new tab)	Hopson, M., McFadden, V., Refoy, R., & Rouault, A. (Eds.). (2020). De-risking government technology: Federal agency field guide. 18F, U.S. General Services Administration. digitalgovernmenthub.org
Moment of truth	A point of contact at which a user forms a lasting judgement of the whole service, such as a first login or a results day.	R02Service design and systems thinkingFrom the field · not in the guide	Carlzon, J. (1987). Moments of truth. Ballinger.
Moment of use	The point at which a person meets a change while doing their task.	R20Communicating change to schoolsThe channels people read · slide 26 (opens in a new tab)	Government Digital Service. (n.d.-b). Help users prepare for change. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk
MoSCoW	A method that splits scope into must, should, could and won't, useful for fixed-date releases, with musts kept to no more than about 60% of effort (Clegg & Barker, 1994).	R01Digital product managementSay no in public · slide 31 (opens in a new tab)	Clegg, D., & Barker, R. (1994). Case method fast-track: A RAD approach. Addison-Wesley. Agile Business Consortium. (n.d.). MoSCoW prioritisation. DSDM Project Framework Handbook. agilebusiness.org
Muda	Japanese for waste: any activity that uses resources without creating value for the user.	R10Lean product developmentThe spine · slide 15 (opens in a new tab)	Ohno, T. (1988). Toyota production system: Beyond large-scale production. Productivity Press. Womack, J. P., & Jones, D. T. (1996). Lean thinking: Banish waste and create wealth in your corporation. Simon & Schuster. lean.org Lean Enterprise Institute. (n.d.). Seven wastes. Lean lexicon. Retrieved September 17, 2026, from lean.org
Multi-armed bandit	An allocation method that shifts traffic towards the better-performing versions while the test runs, trading some certainty about the effect for fewer users on a worse version.	R13Experimentation and product analyticsFrom the field · not in the guide	Lattimore, T., & Szepesvári, C. (2020). Bandit algorithms. Cambridge University Press. doi
Multi-factor authentication (MFA)	Signing in with two or more kinds of proof, such as a password plus a code from a phone, so that a stolen password is not enough.	R17Privacy, security and ethicsFrom the field · not in the guide	Grassi, P. A., Garcia, M. E., & Fenton, J. L. (2017). Digital identity guidelines (NIST Special Publication 800-63-3). National Institute of Standards and Technology. doi
Multimodal learning analytics	Analytics that combines log data with other sources, such as video, audio, gesture, eye tracking or sensor data, to study learning that happens away from a screen.	R15Learning analyticsFrom the field · not in the guide	Blikstein, P. (2013). Multimodal learning analytics. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 102–106). ACM. doi
Multiple comparisons	The problem that running many tests makes it likely some will come out significant by chance alone. Corrections such as Bonferroni's raise the bar for each test.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. doi
N
NASSS framework	A framework for health and care technology programmes that names non-adoption and abandonment as outcomes to plan for, not only to explain afterwards (Greenhalgh et al., 2017).	R19Adoption and change in schoolsPlan the exit · slide 49 (opens in a new tab)	Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017). Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. Journal of Medical Internet Research, 19(11), Article e367. doi
Need	The progress a person is trying to make, inferred from what they say and do.	R04Discovery researchThe spine · slide 13 (opens in a new tab)	Christensen, C. M., Hall, T., Dillon, K., & Duncan, D. S. (2016). Competing against luck: The story of innovation and customer choice. HarperBusiness. archive.org
Negative case	A case that does not fit an emerging pattern. Analysts look for such cases on purpose and revise or limit the pattern until it accounts for them.	R06Design methods IFrom the field · not in the guide	Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. SAGE.
Net Promoter Score	The share of people rating their likelihood to recommend at nine or ten, minus the share rating it six or below (Reichheld, 2003).	R05Evaluative researchScores with observations · slide 46 (opens in a new tab)	Reichheld, F. F. (2003). The one number you need to grow. Harvard Business Review, 81(12), 46–54. hbr.org Keiningham, T. L., Cooil, B., Andreassen, T. W., & Aksoy, L. (2007). A longitudinal examination of Net Promoter and firm revenue growth. Journal of Marketing, 71(3), 39–51. doi
Networked learning community	A group of teachers from different schools who learn through shared practice (Academy of Singapore Teachers, 2026b).	R21Professional learning that changes practiceIn practice, by role · slide 56 (opens in a new tab)	Academy of Singapore Teachers. (2026b). Networked learning communities. Ministry of Education, Singapore. academyofsingaporeteachers.moe.edu.sg Academy of Singapore Teachers. (2026a). Our organisation. Ministry of Education, Singapore. academyofsingaporeteachers.moe.edu.sg
#NoEstimates	A contested practitioner argument for working without estimates, described as exploring alternatives to estimates for making decisions (Zuill, 2013).	R09Agile and Scrum in practiceMake work visible · slide 28 (opens in a new tab)	Zuill, W. (2013, May 17). The NoEstimates hashtag. Life, Liberty, and the Pursuit of Agility. zuill.us
Noise	Anything that distorts or drowns a message between sender and receiver, from a crowded inbox to an inaccurate retelling.	R20Communicating change to schoolsFrom the field · not in the guide	Shannon, C. E., & Weaver, W. (1949). The mathematical theory of communication. University of Illinois Press.
Nominal group	People who generate ideas alone, whose ideas are then pooled.	R03Design thinking, used honestlyA note is a claim · slide 33 (opens in a new tab)	Diehl, M., & Stroebe, W. (1987). Productivity loss in brainstorming groups: Toward the solution of a riddle. Journal of Personality and Social Psychology, 53(3), 497–509. doi Taylor, D. W., Berry, P. C., & Block, C. H. (1958). Does group participation when using brainstorming facilitate or inhibit creative thinking? Administrative Science Quarterly, 3(1), 23–47. doi
Non-functional requirement	A requirement about how well a system works, such as speed, reliability or security, rather than what it does.	R11Software development for non-engineersReading a proposal · slide 43 (opens in a new tab)	ISO/IEC. (2023). Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model (ISO/IEC Standard No. 25010:2023). International Organization for Standardization. iso.org
Nonresponse bias	Error that arises when the people who answer a survey differ from those who do not.	R05Evaluative researchQuestions that cannot lead · slide 33 (opens in a new tab)	Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189. doi Davern, M. (2013). Nonresponse rates are a problematic indicator of nonresponse bias in survey research. Health Services Research, 48(3), 905–912. doi
Normal distribution	The symmetrical, bell-shaped distribution in which most values sit near the mean and extreme values are rare. Many common tests assume it.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Wheelan, C. (2013). Naked statistics: Stripping the dread from the data. W. W. Norton.
North star	A product team's long-term objective. In a learning product, usually a learning outcome that takes too long to measure in one test.	R13Experimentation and product analyticsThe metric first · slide 19 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
North star metric	One metric chosen to represent the value a product delivers, supported by input metrics a team can move.	R01Digital product managementThe leading signal · slide 43 (opens in a new tab)	Amplitude. (2019). The North Star playbook. amplitude.com
Novelty effect	A temporary change in behaviour because something is new, which fades as users get used to it (Kohavi et al., 2012).	R13Experimentation and product analyticsNo peeking · slide 46 (opens in a new tab)	Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Xu, Y. (2012). Trustworthy online controlled experiments: Five puzzling outcomes explained. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 786–794). Association for Computing Machinery. doi Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459. doi Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
Now, next, later roadmap	A roadmap with three time horizons instead of dates: work under way, work coming soon and work further out, with less detail and less certainty in each later column.	R01Digital product managementFrom the field · not in the guide	Lombardo, C. T., McCarthy, B., Ryan, E., & Connors, M. (2017). Product roadmaps relaunched: How to set direction while embracing uncertainty. O'Reilly Media.
Nudge	A small prompt or change in how choices are presented, such as a reminder message, meant to steer behaviour without removing options.	R15Learning analyticsFrom the field · not in the guide	Thaler, R. H., & Sunstein, C. R. (2008). Nudge: Improving decisions about health, wealth, and happiness. Yale University Press.
Nugget	A tagged observation supported by evidence, the smallest unit of atomic research (Sharon, 2018).	R06Design methods IKeep the trail · slide 27 (opens in a new tab)	Sharon, T. (2016). The atomic unit of a research insight. Medium. tsharon.medium.com Sharon, T. (2018). Foundations of atomic research. Medium. tsharon.medium.com
Null hypothesis	The assumption of no effect that a significance test measures data against.	R18Basic statistics for educational technologistsUncertainty · slide 25 (opens in a new tab)	Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. doi Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. doi
O
Objectives and key results (OKRs)	A goal-setting format that pairs a qualitative objective with two to five measurable results showing whether it was reached, usually set each quarter.	R01Digital product managementFrom the field · not in the guide	Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley. Doerr, J. (2018). Measure what matters: How Google, Bono, and the Gates Foundation rock the world with OKRs. Portfolio/Penguin. Grove, A. S. (1983). High output management. Random House.
Observability (of a system)	How well a team can work out what is happening inside a running system from the logs, metrics and traces it emits, including for problems nobody predicted.	R12DevOps and continuous deliveryFrom the field · not in the guide	Majors, C., Fong-Jones, L., & Miranda, G. (2022). Observability engineering: Achieving production excellence. O'Reilly Media.
Observability (of an innovation)	How visible the results of an innovation are to others. Teachers adopt more readily what they can see working in a colleague's classroom.	R19Adoption and change in schoolsFrom the field · not in the guide	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Observation	A record of what happened, of the kind a camera would capture. R06 Something that happened in a research session and that others could check from the record.	R04Discovery researchNotes and meanings · slide 36 (opens in a new tab) R06Design methods IThe spine · slide 13 (opens in a new tab)	Ross, R. (1994). The ladder of inference. In P. M. Senge, A. Kleiner, C. Roberts, R. B. Ross, & B. J. Smith, The fifth discipline fieldbook: Strategies and tools for building a learning organization. Currency Doubleday. archive.org Pidcock, D. (2018). What is atomic UX research? Prototypr. blog.prototypr.io Beyer, H., & Holtzblatt, K. (1998). Contextual design: Defining customer-centered systems. Morgan Kaufmann. archive.org
On-call	A rota under which a named engineer is reachable for a set period and must respond within an agreed time when the live system raises an alert.	R12DevOps and continuous deliveryFrom the field · not in the guide	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
OneRoster	An education data standard for exchanging roster information, course materials and grades between systems. It answers who is in which class (1EdTech Consortium, n.d.-b).	R14Data strategy and governanceIdentifiers first · slide 39 (opens in a new tab)	1EdTech Consortium. (n.d.-b). OneRoster. Retrieved September 17, 2026, from 1edtech.org
Open data	Data published for anyone to access, use and share freely, usually under an open licence and in a machine-readable format.	R14Data strategy and governanceFrom the field · not in the guide	OECD. (2019). The path to becoming a data-driven public sector. OECD Publishing. doi
Open question	A question that cannot be answered with yes, no or a single word, so the participant has to describe or explain in their own words.	R04Discovery researchFrom the field · not in the guide	Rosala, M. (2022). 6 mistakes when crafting interview questions. Nielsen Norman Group. nngroup.com Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com
Open rate	The share of delivered emails that recipients opened. It is a rough measure, because image blocking and privacy features distort the count.	R20Communicating change to schoolsFrom the field · not in the guide
Open-source software	Software whose source code is published under a licence that lets anyone use, study, change and redistribute it. Most modern systems are built largely from open-source components.	R11Software development for non-engineersFrom the field · not in the guide	Raymond, E. S. (1999). The cathedral and the bazaar: Musings on Linux and open source by an accidental revolutionary. O'Reilly Media. doi
Opinion leader	A member of a group whose views colleagues seek out and follow. Their take-up of a tool, or their refusal, strongly influences everyone else's.	R19Adoption and change in schoolsFrom the field · not in the guide	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Opportunity	A need, pain point or desire that, if addressed, could move an outcome. R06 An insight turned into something to design for, usually written as a question.	R01Digital product managementName the outcome first · slide 19 (opens in a new tab) R06Design methods IThe spine · slide 13 (opens in a new tab)	Torres, T. (2021). Continuous discovery habits: Discover products that create customer value and business value. Product Talk. IDEO.org. (2015). The field guide to human-centered design. IDEO.org. designkit.org Rosala, M. (2021). Using "How might we" questions to ideate on the right problems. Nielsen Norman Group. nngroup.com
Opportunity solution tree	A map with the desired outcome at its root, the opportunity space of needs and pain points below it, then the solutions, each carrying small tests of its riskiest assumptions (Torres, 2021).	R01Digital product managementName the outcome first · slide 19 (opens in a new tab)	Torres, T. (2021). Continuous discovery habits: Discover products that create customer value and business value. Product Talk.
Optional stopping	Checking a running test repeatedly and stopping when the result looks good. It inflates false positives.	R13Experimentation and product analyticsNo peeking · slide 45 (opens in a new tab)	Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2022). Always valid inference: Continuous monitoring of A/B tests. Operations Research, 70(3), 1806–1821. doi Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
OU Analyse	The Open University’s system that predicts which students are at risk of failing their next assignment, with predictions updated weekly for course tutors and student support teams (The Open University, n.d.).	R15Learning analyticsThe action first · slide 26 (opens in a new tab)	The Open University. (n.d.). OU Analyse. Retrieved September 17, 2026, from research.stem.open.ac.uk Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z., Vaclavek, J., & Wolff, A. (2015). OU Analyse: Analysing at-risk students at The Open University. Learning Analytics Review, LAK15-1, 1–16. oro.open.ac.uk Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019). A large-scale implementation of predictive learning analytics in higher education: The teachers’ role and perspective. Educational Technology Research and Development, 67, 1273–1306. doi
Outcome	A change in someone's behaviour that leads towards the result you want.	R01Digital product managementThe spine · slide 12 (opens in a new tab)	Seiden, J. (2019). Outcomes over output: Why customer behavior is the key metric for business success. Sense & Respond Press. W. K. Kellogg Foundation. (2004). Logic model development guide. naccho.org Perri, M. (2018). Escaping the build trap: How effective product management creates real value. O'Reilly Media.
Outlier	A value far from the rest of the data. It may be an error or a real and important case, so it should be investigated before it is removed.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley. doi
Output	Something a team produces: a feature, a release, a document.	R01Digital product managementThe spine · slide 13 (opens in a new tab)	W. K. Kellogg Foundation. (2004). Logic model development guide. naccho.org Seiden, J. (2019). Outcomes over output: Why customer behavior is the key metric for business success. Sense & Respond Press. Perri, M. (2018). Escaping the build trap: How effective product management creates real value. O'Reilly Media.
Overall evaluation criterion	The primary measure a test is designed to move, chosen in advance (Kohavi et al., 2020).	R13Experimentation and product analyticsThe metric first · slide 19 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., & Xu, Y. (2012). Trustworthy online controlled experiments: Five puzzling outcomes explained. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 786–794). Association for Computing Machinery. doi
Overfitting	A model that captures the quirks of past data performs well in-sample and fails on new data. Only out-of-sample performance tests prediction honestly (Hastie et al., 2009).	R18Basic statistics for educational technologistsWhere numbers come from · slide 53 (opens in a new tab)	Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.
OWASP Top 10	A regularly updated list of the most serious security risks in web applications, such as broken access control and injection, widely used as a baseline in procurement and testing.	R17Privacy, security and ethicsFrom the field · not in the guide	OWASP Foundation. (2021). OWASP Top 10:2021. OWASP Foundation.
Owner	The one person accountable for a dataset: who may use it, what quality it needs and when it ends.	R14Data strategy and governanceNamed owners · slide 24 (opens in a new tab)	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org Privacy Technical Assistance Center. (2015). Data governance and stewardship (Rev. ed.). U.S. Department of Education. studentprivacy.ed.gov OECD. (2019). The path to becoming a data-driven public sector. OECD Publishing. doi
P
p-hacking	Running many analyses on the same data, which raises the odds that one crosses 0.05 by chance (Simmons et al., 2011).	R18Basic statistics for educational technologistsUncertainty · slide 27 (opens in a new tab)	Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi
p-value	The probability of data at least this extreme if there were no real effect.	R18Basic statistics for educational technologistsUncertainty · slide 25 (opens in a new tab)	Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. doi Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. doi Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi
Pain point	A specific place in a task or journey where a person meets difficulty, delay or frustration.	R06Design methods IFrom the field · not in the guide	Gibbons, S. (2018). Journey mapping 101. Nielsen Norman Group. nngroup.com Kalbach, J. (2020). Mapping experiences: A complete guide to customer alignment through journeys, blueprints, and diagrams (2nd ed.). O'Reilly Media. openlibrary.org
Pair programming	Two developers working at one computer on the same code, one writing while the other reviews, and swapping roles often.	R09Agile and Scrum in practiceFrom the field · not in the guide	Beck, K., & Andres, C. (2004). Extreme programming explained: Embrace change (2nd ed.). Addison-Wesley. Williams, L., & Kessler, R. (2002). Pair programming illuminated. Addison-Wesley.
Paper prototype	Hand-drawn screens moved by a facilitator in response to what a user does.	R07Design methods IIMatch the fidelity · slide 26 (opens in a new tab)	Rudd, J., Stern, K., & Isensee, S. (1996). Low vs. high-fidelity prototyping debate. Interactions, 3(1), 76–85. doi Walker, M., Takayama, L., & Landay, J. A. (2002). High-fidelity or low-fidelity, paper or computer? Choosing attributes when testing web prototypes. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 46(5), 661–665. doi
Parallel prototyping	Making several prototypes before getting feedback on them together (Dow et al., 2010).	R07Design methods IISketch many · slide 20 (opens in a new tab)	Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010). Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction, 17(4), Article 18. doi Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006). Getting the right design and the design right: Testing many is better than one. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1243–1252). ACM. doi
Paraphrase test	A test in which readers restate a text in their own words, to show what they understood (General Services Administration, n.d.).	R20Communicating change to schoolsEquip the middle · slide 46 (opens in a new tab)	General Services Administration. (n.d.). Paraphrase testing. Digital.gov plain language guide series. Retrieved September 17, 2026, from digital.gov
Participant observation	Fieldwork in which the researcher joins in the activity being studied, to some degree, while observing it.	R04Discovery researchFrom the field · not in the guide	Spradley, J. P. (1980). Participant observation. Holt, Rinehart and Winston. doi
Participatory design	A tradition, begun in Scandinavian workplaces, in which the people who will use a system take part in designing it and share in the decisions about it.	R03Design thinking, used honestlyFrom the field · not in the guide	Sanders, E. B.-N., & Stappers, P. J. (2008). Co-creation and the new landscapes of design. CoDesign, 4(1), 5–18. doi Schuler, D., & Namioka, A. (Eds.). (1993). Participatory design: Principles and practices. Lawrence Erlbaum Associates.
Pattern	An observation that repeats across people or sessions, reported with a count. R08 A reusable solution to a common user task, such as signing in, built from components.	R06Design methods IThe spine · slide 13 (opens in a new tab) R08UX foundations and design systemsSystem first · slide 31 (opens in a new tab)	Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. doi Frost, B. (2016). Atomic design. Brad Frost Web. atomicdesign.bradfrost.com Government Digital Service. (n.d.-a). Contribution criteria. GOV.UK Design System. Retrieved September 17, 2026, from design-system.service.gov.uk
Pattern library	A catalogue of an organisation's interface components and patterns, with examples and guidance on use. It is one part of a design system.	R08UX foundations and design systemsFrom the field · not in the guide	Frost, B. (2016). Atomic design. Brad Frost Web. atomicdesign.bradfrost.com Kholmatova, A. (2017). Design systems: A practical guide to creating design languages for digital products. Smashing Media.
PDSA cycle	Plan, Do, Study, Act: the improvement cycle used in improvement science in education.	R10Lean product developmentCount learning · slide 40 (opens in a new tab)	Bryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How America's schools can get better at getting better. Harvard Education Press. books.google.com Carnegie Foundation for the Advancement of Teaching. (n.d.). The six core principles of improvement. Retrieved September 17, 2026, from carnegiefoundation.org
Pedagogical content knowledge (PCK)	A teacher's knowledge of how to make a particular subject understandable: its useful representations, and the difficulties and misconceptions students bring to it.	R21Professional learning that changes practiceFrom the field · not in the guide	Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. doi Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. doi
Peer coaching	Teachers working in coaching groups to carry training into classroom use; the later model dropped peer feedback, and teachers planned together and observed each other instead (Showers & Joyce, 1996).	R21Professional learning that changes practiceCoaching · slide 35 (opens in a new tab)	Showers, B., & Joyce, B. (1996). The evolution of peer coaching. Educational Leadership, 53(6), 12–16. ascd.org Joyce, B., & Showers, B. (2002). Student achievement through staff development (3rd ed.). Association for Supervision and Curriculum Development. books.google.com
Penetration test	An authorised attack on a system by specialists, to find weaknesses before a real attacker does; often required before launch or in a contract.	R17Privacy, security and ethicsFrom the field · not in the guide	Scarfone, K., Souppaya, M., Cody, A., & Orebaugh, A. (2008). Technical guide to information security testing and assessment (NIST Special Publication 800-115). National Institute of Standards and Technology. doi
Perceived ease of use	How free of effort a person expects a system to be (Davis, 1989).	R19Adoption and change in schoolsEffort before training · slide 22 (opens in a new tab)	Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. doi Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. doi
Perceived usefulness	How much a person believes a system will help their work (Davis, 1989).	R19Adoption and change in schoolsEffort before training · slide 22 (opens in a new tab)	Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. doi Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. doi
Percentile	The value below which a given percentage of results fall. A student at the 80th percentile scored higher than 80 per cent of the group.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Wheelan, C. (2013). Naked statistics: Stripping the dread from the data. W. W. Norton.
Permissioned opinions	The fourth step of the Critical Response Process, in which opinions come last and only with the maker's permission: "I have an opinion about _____, would you like to hear it?" (Lerman, n.d.).	R07Design methods IICritique against the goal · slide 34 (opens in a new tab)	Lerman, L. (n.d.). Critical Response Process. Liz Lerman. Retrieved September 18, 2026, from lizlerman.com
Persona	A description of a fictional user that stands for a group. A model to test, not a finding.	R06Design methods IPersonas as hypotheses · slide 32 (opens in a new tab)	Cooper, A. (1999). The inmates are running the asylum: Why high-tech products drive us crazy and how to restore the sanity. Sams. archive.org Pruitt, J., & Grudin, J. (2003). Personas: Practice and theory. In Proceedings of the 2003 Conference on Designing for User Experiences (pp. 1–15). ACM. doi Chapman, C. N., & Milham, R. P. (2006). The personas' new clothes: Methodological and practical arguments against a popular method. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(5), 634–636. doi
Personal data	Any information about an identified or identifiable living person, including a student number, device identifier or learning record that can be linked to a student. United States usage is personally identifiable information (PII).	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Personal Data Protection Act 2012 (2020 Rev. Ed.) (Singapore). McCallister, E., Grance, T., & Scarfone, K. (2010). Guide to protecting the confidentiality of personally identifiable information (PII) (NIST Special Publication 800-122). National Institute of Standards and Technology. doi
Personal Data Protection Act (PDPA)	Singapore's 2012 data protection law for private organisations, including edtech vendors. Government agencies, and so government schools, follow separate public sector rules.	R17Privacy, security and ethicsFrom the field · not in the guide	Personal Data Protection Act 2012 (2020 Rev. Ed.) (Singapore).
Phishing	A message that poses as a trusted sender to trick someone into giving a password, opening a file or paying a false invoice. Aimed at one named person, it is spear phishing.	R17Privacy, security and ethicsFrom the field · not in the guide	Jagatic, T. N., Johnson, N. A., Jakobsson, M., & Menczer, F. (2007). Social phishing. Communications of the ACM, 50(10), 94–100. doi Mitnick, K. D., & Simon, W. L. (2002). The art of deception: Controlling the human element of security. Wiley.
Physical evidence	The tangible things a user meets at each step of a service, such as a letter, a screen, a form or a room. It forms the top row of a service blueprint.	R02Service design and systems thinkingFrom the field · not in the guide	Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008). Service blueprinting: A practical technique for service innovation. California Management Review, 50(3), 66–94. doi Shostack, G. L. (1984). Designing services that deliver. Harvard Business Review, 62(1), 133–139. hbr.org
Pilot	A small, time-limited trial of a tool or practice in a few schools or classes, run to learn whether and how it works before a wider rollout.	R19Adoption and change in schoolsFrom the field · not in the guide	Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk
Pilot test	A trial run of a study with one or two people, held to find faults in the tasks, questions, timing and equipment before the real sessions begin.	R05Evaluative researchFrom the field · not in the guide	Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Wiley.
Pivot	A structured change of course to test a new fundamental hypothesis (Ries, 2011).	R10Lean product developmentCount learning · slide 38 (opens in a new tab)	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com Sadeghiani, A., & Anderson, A. R. (2023). What pivot is: Touching an elephant in the dark. Digital Business, 3(1), Article 100056. doi Camuffo, A., Gambardella, A., Messinese, D., Novelli, E., Paolucci, E., & Spina, C. (2024). A scientific approach to entrepreneurial decision-making: Large-scale replication and extension. Strategic Management Journal, 45(6), 1209–1237. doi
Placeholder text	Dummy wording, often the mock-Latin lorem ipsum, used to fill a layout before the real content exists. It hides the problems real content would reveal.	R07Design methods IIFrom the field · not in the guide	Richards, S. (2017). Content design. Content Design London. contentdesign.london
Plain language	Writing that lets its intended readers find what they need, understand it and use it (International Organization for Standardization, 2023).	R20Communicating change to schoolsLead with the teacher · slide 21 (opens in a new tab)	International Organization for Standardization. (2023). Plain language: Part 1. Governing principles and guidelines (ISO Standard No. 24495-1:2023). iso.org Government Digital Service. (n.d.-d). Use clear language. GOV.UK content and publishing guidance. Retrieved September 17, 2026, from guidance.publishing.service.gov.uk General Services Administration. (n.d.). Paraphrase testing. Digital.gov plain language guide series. Retrieved September 17, 2026, from digital.gov
Planning fallacy	The tendency to forecast from the details of one plan and underestimate time and cost.	R09Agile and Scrum in practiceThe school calendar · slide 40 (opens in a new tab)	Kahneman, D., & Tversky, A. (1979). Intuitive prediction: Biases and corrective procedures. In S. Makridakis & S. C. Wheelwright (Eds.), Forecasting (TIMS Studies in the Management Sciences, Vol. 12, pp. 313–327). North-Holland. Flyvbjerg, B. (2006). From Nobel Prize to project management: Getting risks right. Project Management Journal, 37(3), 5–15. doi
Planning poker	An estimating technique in which team members reveal their estimates for a piece of work at the same moment, using numbered cards, then discuss the differences and repeat until they converge.	R09Agile and Scrum in practiceFrom the field · not in the guide	Cohn, M. (2005). Agile estimating and planning. Prentice Hall.
Platform team	A team that provides shared tools and services, such as the deployment pipeline, hosting and monitoring, as an internal product, so that product teams can release without building these themselves.	R12DevOps and continuous deliveryFrom the field · not in the guide	Skelton, M., & Pais, M. (2019). Team topologies: Organizing business and technology teams for fast flow. IT Revolution Press.
Policy resistance	The tendency for interventions to be defeated by the system's response to them (Sterman, 2006).	R02Service design and systems thinkingThe spine · slide 15 (opens in a new tab)	Sterman, J. D. (2006). Learning from evidence in a complex world. American Journal of Public Health, 96(3), 505–514. doi Merton, R. K. (1936). The unanticipated consequences of purposive social action. American Sociological Review, 1(6), 894–904. doi Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green.
Population	The whole group a conclusion is meant to apply to.	R18Basic statistics for educational technologistsWhat statistics is for · slide 8 (opens in a new tab)	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican.
Postmortem	A written record of an incident, its impact, the actions taken, its causes and the follow-up to prevent it recurring (Lunney & Lueder, 2016).	R22Support and feedback loopsTime to fix · slide 41 (opens in a new tab)	Lunney, J., & Lueder, S. (2016). Postmortem culture: Learning from failure. In B. Beyer, C. Jones, J. Petoff, & N. R. Murphy (Eds.), Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Power	The probability a study detects an effect of a given size if it truly exists.	R18Basic statistics for educational technologistsUncertainty · slide 27 (opens in a new tab)	Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. doi
Power and interest grid	A grid that sorts stakeholders by how much they can affect a decision and how much they care about it (Mendelow, 1981).	R01Digital product managementWork back from the mandate · slide 38 (opens in a new tab)	Mendelow, A. L. (1981). Environmental scanning: The impact of the stakeholder concept. In Proceedings of the Second International Conference on Information Systems. Association for Information Systems. aisel.aisnet.org
Pre-registration	A dated record of a test's outcome, analysis and decision rule, made before the data arrives.	R13Experimentation and product analyticsWrite it down first · slide 31 (opens in a new tab)	Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi Education Endowment Foundation. (2022). Statistical analysis guidance for EEF evaluations. d2tic4wvo1iusb.cloudfront.net Inter-university Consortium for Political and Social Research. (n.d.). Registry of Efficacy and Effectiveness Studies (REES). Retrieved September 17, 2026, from icpsr.umich.edu
Precision	Of the students flagged, the share who really had the outcome.	R15Learning analyticsThe base rate · slide 40 (opens in a new tab)	Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. doi
Predictive analytics	The use of past data to estimate the probability of a future outcome for each student, such as failing a module or withdrawing.	R15Learning analyticsFrom the field · not in the guide	Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019). A large-scale implementation of predictive learning analytics in higher education: The teachers’ role and perspective. Educational Technology Research and Development, 67, 1273–1306. doi Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. doi
Pretotype	A mock-up or pretence of a product, cheaper than a prototype, used to test whether people would want and use it before testing whether it can be built.	R10Lean product developmentFrom the field · not in the guide	Savoia, A. (2019). The right it: Why so many ideas fail and how to make sure yours succeed. HarperOne.
Primacy effect	A short-lived dip in a metric after a change, because experienced users need time to adjust to it. The opposite of the novelty effect.	R13Experimentation and product analyticsFrom the field · not in the guide	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
Prime directive	An opening rule for retrospectives, that everyone did the best job they could, given what they knew at the time, which keeps the conversation on the process rather than on blame (Kerth, 2001).	R09Agile and Scrum in practiceInspect the process · slide 33 (opens in a new tab)	Kerth, N. L. (2001). Project retrospectives: A handbook for team reviews. Dorset House. Derby, E., & Larsen, D. (2006). Agile retrospectives: Making good teams great. Pragmatic Bookshelf. pragprog.com
Privacy by design	Building privacy protection into a system from the first design decision, with the most protective settings as the default, instead of adding it after the build.	R17Privacy, security and ethicsFrom the field · not in the guide	Cavoukian, A. (2009). Privacy by design: The 7 foundational principles. Information and Privacy Commissioner of Ontario. European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88.
Privacy notice	The plain statement given to students and parents of what data is collected, why, who receives it, how long it is kept and what rights they have.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Information Commissioner's Office. (2020). Age appropriate design: A code of practice for online services. Information Commissioner's Office.
Probe	A follow-up question that asks a participant to say more about something they have just said, such as "what happened next?"	R04Discovery researchFrom the field · not in the guide	Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com Spradley, J. P. (1979). The ethnographic interview. Holt, Rinehart and Winston.
Problem	A cause, or potential cause, of one or more incidents (AXELOS, 2019).	R22Support and feedback loopsThe spine · slide 14 (opens in a new tab)	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com Flora, E. (2023b). An overview of the problem management practice in ITIL 4. Beyond20. beyond20.com
Problem setting	Naming what to attend to and framing the context around it (Schön, 1983).	R03Design thinking, used honestlyThe spine · slide 13 (opens in a new tab)	Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Basic Books. Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. doi
Problem space and solution space	The problem space is everything a team could learn about the need and its setting; the solution space is everything it could build in response. Design thinking explores the first before the second.	R03Design thinking, used honestlyFrom the field · not in the guide	Newell, A., & Simon, H. A. (1972). Human problem solving. Prentice-Hall. doi Dorst, K., & Cross, N. (2001). Creativity in the design process: Co-evolution of problem–solution. Design Studies, 22(5), 425–437. doi
Problem statement	A short statement of who has a problem, what the problem is and why it matters, written without proposing a solution.	R06Design methods IFrom the field · not in the guide	Rosala, M. (2021). Using "How might we" questions to ideate on the right problems. Nielsen Norman Group. nngroup.com IDEO.org. (2015). The field guide to human-centered design. IDEO.org. designkit.org
Product	Something maintained and improved over time by a stable team, judged by the outcomes it moves.	R01Digital product managementThe misconception · slide 10 (opens in a new tab)	Kersten, M. (2018). Project to product: How to survive and thrive in the age of digital disruption with the Flow Framework. IT Revolution. Cagan, M., Hickman, L., Jones, C., Idiodi, C., & Moore, J. (2024). Transformed: Moving to the product operating model. Wiley. Thomas, D. (2018, October 29). Funding product teams, not projects. Defra Digital. defradigital.blog.gov.uk
Product life cycle	The stages a product passes through from introduction, through growth and maturity, to decline and retirement, each calling for different investment and different measures.	R01Digital product managementFrom the field · not in the guide	Levitt, T. (1965). Exploit the product life cycle. Harvard Business Review, 43(6), 81–94.
Product manager	The person answerable for what a product team builds and why, who weighs user value against cost and the organisation's aims. Distinct from a project manager, who is answerable for schedule and budget.	R01Digital product managementFrom the field · not in the guide	Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley. Perri, M. (2018). Escaping the build trap: How effective product management creates real value. O'Reilly Media.
Product operating model	A way of organising technology work around long-lived teams that are given problems to solve and judged by outcomes, in place of projects that are handed features to deliver.	R01Digital product managementFrom the field · not in the guide	Cagan, M., Hickman, L., Jones, C., Idiodi, C., & Moore, J. (2024). Transformed: Moving to the product operating model. Wiley. Kersten, M. (2018). Project to product: How to survive and thrive in the age of digital disruption with the Flow Framework. IT Revolution.
Product operations (product ops)	The function that supports product managers with shared data, user-research logistics, tooling and common processes, so that teams spend their time on decisions.	R01Digital product managementFrom the field · not in the guide	Cagan, M., Hickman, L., Jones, C., Idiodi, C., & Moore, J. (2024). Transformed: Moving to the product operating model. Wiley. Perri, M., & Tilles, D. (2023). Product operations: How successful companies build better products at scale. Product Thinking.
Product Owner	The one person in a Scrum team accountable for the value of the product, who orders the Product Backlog and decides what the team works on next.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Product quality model	The international model of nine characteristics of software quality, among them functional suitability, reliability, security, maintainability and safety (ISO/IEC, 2023).	R11Software development for non-engineersReading a proposal · slide 43 (opens in a new tab)	ISO/IEC. (2023). Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model (ISO/IEC Standard No. 25010:2023). International Organization for Standardization. iso.org
Product requirements document (PRD)	A document that sets out the problem a feature should solve, who it is for, what it must do and how success will be judged, written before the build begins.	R01Digital product managementFrom the field · not in the guide	Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley.
Product strategy	The sequence of problems a team chooses to solve, and those it chooses not to, in order to move from where the product is now towards its vision.	R01Digital product managementFrom the field · not in the guide	Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley. Perri, M. (2018). Escaping the build trap: How effective product management creates real value. O'Reilly Media.
Product trio	The product, design and engineering leads who run discovery together (Torres, 2021).	R01Digital product managementName the outcome first · slide 19 (opens in a new tab)	Torres, T. (2021). Continuous discovery habits: Discover products that create customer value and business value. Product Talk.
Product vision	A short description of the future the product is meant to bring about for its users, usually three to ten years out, which gives the roadmap its direction.	R01Digital product managementFrom the field · not in the guide	Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley. Lombardo, C. T., McCarthy, B., Ryan, E., & Connors, M. (2017). Product roadmaps relaunched: How to set direction while embracing uncertainty. O'Reilly Media.
Product–market fit	The point at which a product meets a real need for a defined group well enough that they keep using it, recommend it and would miss it if it went.	R01Digital product managementFrom the field · not in the guide	Cagan, M. (2018). Inspired: How to create tech products customers love (2nd ed.). Wiley. Olsen, D. (2015). The lean product playbook: How to innovate with minimum viable products and rapid customer feedback. Wiley. doi
Production blocking	The loss of ideas in a group because only one person can speak at a time (Diehl & Stroebe, 1987).	R03Design thinking, used honestlyA note is a claim · slide 33 (opens in a new tab)	Diehl, M., & Stroebe, W. (1987). Productivity loss in brainstorming groups: Toward the solution of a riddle. Journal of Personality and Social Psychology, 53(3), 497–509. doi
Production environment	The live system that real users rely on, as distinct from the environments used for development and testing. Often shortened to 'production' or 'prod'.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Productive failure	A design in which students attempt problems before instruction, and learn from the attempt (Kapur, 2008).	R16AI in learning productsProtect the struggle · slide 26 (opens in a new tab)	Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–424. doi Sinha, T., & Kapur, M. (2021). When problem solving followed by instruction works: Evidence for productive failure. Review of Educational Research, 91(5), 761–798. doi
Productive struggle	Effort on the thinking a lesson exists to build, as distinct from effort wasted on confusion.	R16AI in learning productsProtect the struggle · slide 24 (opens in a new tab)	Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–424. doi Sinha, T., & Kapur, M. (2021). When problem solving followed by instruction works: Evidence for productive failure. Review of Educational Research, 91(5), 761–798. doi Ministry of Education, Singapore. (2026). Artificial intelligence in education (Last updated July 31, 2026). moe.gov.sg
Professional development	Structured, facilitated activity for teachers intended to increase their teaching ability (Sims et al., 2021).	R21Professional learning that changes practiceThe second week · slide 47 (opens in a new tab)	Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov Desimone, L. M. (2009). Improving impact studies of teachers' professional development: Toward better conceptualizations and measures. Educational Researcher, 38(3), 181–199. doi Kennedy, M. M. (2016). How does professional development improve teaching? Review of Educational Research, 86(4), 945–980. doi
Professional learning community (PLC)	A group of teachers in a school who meet regularly to examine student learning, try changes to their teaching and review the results together.	R21Professional learning that changes practiceFrom the field · not in the guide	Stoll, L., Bolam, R., McMahon, A., Wallace, M., & Thomas, S. (2006). Professional learning communities: A review of the literature. Journal of Educational Change, 7(4), 221–258. doi DuFour, R., & Eaker, R. (1998). Professional learning communities at work: Best practices for enhancing student achievement. National Educational Service.
Progressive disclosure	Giving help in increasing steps, with a full solution last and only after an attempt.	R16AI in learning productsProtect the struggle · slide 25 (opens in a new tab)	Department for Education. (2026). Generative AI: Product safety standards (Updated January 19, 2026). GOV.UK. gov.uk Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), Article e2422633122. doi
Project	Temporary work to deliver a defined scope by a date and within a budget.	R01Digital product managementThe misconception · slide 10 (opens in a new tab)	Kersten, M. (2018). Project to product: How to survive and thrive in the age of digital disruption with the Flow Framework. IT Revolution. Thomas, D. (2018, October 29). Funding product teams, not projects. Defra Digital. defradigital.blog.gov.uk HM Treasury, Department for Science, Innovation and Technology, & Government Digital Service. (2025, March 12). Performance review of digital spend. GOV.UK. gov.uk
Prompt	The instruction and material given to a generative model as input. The model's output depends on it, so wording, examples and supplied context all matter.	R16AI in learning productsFrom the field · not in the guide	Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), Article 195. doi
Proof of concept	A small build made to show that an idea is technically possible, which says little about whether people want it or can use it.	R07Design methods IIFrom the field · not in the guide	Houde, S., & Hill, C. (1997). What do prototypes prototype? In M. Helander, T. K. Landauer, & P. Prabhu (Eds.), Handbook of human-computer interaction (2nd ed., pp. 367–381). Elsevier. doi
Propensity score matching	A method that pairs participants and non-participants with a similar estimated probability of taking part, given their observed characteristics, to make the groups more comparable.	R13Experimentation and product analyticsFrom the field · not in the guide	Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. doi
Proto persona	A persona built from a team's assumptions, with no new research (Laubheimer, 2020).	R06Design methods IThe misconception · slide 9 (opens in a new tab)	Laubheimer, P. (2020). 3 persona types: Lightweight, qualitative, and statistical. Nielsen Norman Group. nngroup.com Gothelf, J., & Seiden, J. (2013). Lean UX: Applying lean principles to improve user experience. O'Reilly Media.
Prototype	Any representation of a design idea, regardless of medium, made to answer a question (Houde & Hill, 1997). R07 Any representation of a design idea, regardless of medium, built to answer a question and then be thrown away (Houde & Hill, 1997).	R03Design thinking, used honestlyPrototype to learn · slide 26 (opens in a new tab) R07Design methods IIThe misconception · slide 8 (opens in a new tab)	Houde, S., & Hill, C. (1997). What do prototypes prototype? In M. G. Helander, T. K. Landauer, & P. V. Prabhu (Eds.), Handbook of human-computer interaction (2nd ed., pp. 367–381). North-Holland. doi Houde, S., & Hill, C. (1997). What do prototypes prototype? In M. Helander, T. K. Landauer, & P. Prabhu (Eds.), Handbook of human-computer interaction (2nd ed., pp. 367–381). Elsevier. doi Buxton, B. (2007). Sketching user experiences: Getting the design right and the right design. Morgan Kaufmann. books.google.com
Proxy	A measure that stands in for something harder to observe, such as time online standing in for effort.	R15Learning analyticsActivity is not learning · slide 19 (opens in a new tab)	Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. doi Strathern, M. (1997). ‘Improving ratings’: Audit in the British university system. European Review, 5(3), 305–321. cambridge.org Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. doi
Pseudonymisation	Replacing names and other identifiers with codes, with the key held separately. The data remains personal data, because whoever holds the key can reverse it.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. Article 29 Data Protection Working Party. (2014). Opinion 05/2014 on anonymisation techniques (WP 216). European Commission.
Pull request	A proposal to merge a set of changes from a branch into the main code, which teammates can discuss, review and test before it is accepted. Some tools call it a merge request.	R11Software development for non-engineersFrom the field · not in the guide	Chacon, S., & Straub, B. (2014). Pro Git (2nd ed.). Apress. Gousios, G., Pinzger, M., & van Deursen, A. (2014). An exploratory study of the pull-based software development model. In Proceedings of the 36th International Conference on Software Engineering (pp. 345–355). ACM. doi
Pull system	A way of scheduling work in which a step starts a new item only when the step after it signals it has capacity, instead of work being pushed forward to a plan.	R10Lean product developmentFrom the field · not in the guide	Womack, J. P., & Jones, D. T. (1996). Lean thinking: Banish waste and create wealth in your corporation. Simon & Schuster. lean.org Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Blue Hole Press. books.google.com
Purpose limitation	Collecting data for specified, explicit purposes and not using it in ways incompatible with them (European Union, 2016).	R14Data strategy and governanceThe decision first · slide 21 (opens in a new tab)	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. eur-lex.europa.eu Personal Data Protection Commission. (n.d.). Data protection obligations. Retrieved September 17, 2026, from pdpc.gov.sg
Purposive sampling	Choosing participants deliberately because they can say most about the question, not at random. It is the usual approach in qualitative studies.	R04Discovery researchFrom the field · not in the guide	Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). SAGE.
Pushed-down work	Effort a change moves onto people further down the line, usually in schools.	R02Service design and systems thinkingThe pushed-down work · slide 30 (opens in a new tab)	Herd, P., & Moynihan, D. P. (2018). Administrative burden: Policymaking by other means. Russell Sage Foundation. doi Selwyn, N., Nemorin, S., & Johnson, N. F. (2017). High-tech, hard work: An investigation of teachers' work in the digital age. Learning, Media and Technology, 42(4), 390–405. doi
Q
Qualitative and quantitative research	Qualitative methods collect data directly and suit questions about why and how; quantitative methods collect it indirectly and suit questions about how many and how much (Rohrer, 2022).	R04Discovery researchWatch first · slide 19 (opens in a new tab)	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com
Qualitative persona	A persona built from qualitative research, starting from interviews with 5 to 30 users (Laubheimer, 2020).	R06Design methods IThe misconception · slide 9 (opens in a new tab)	Laubheimer, P. (2020). 3 persona types: Lightweight, qualitative, and statistical. Nielsen Norman Group. nngroup.com Pruitt, J., & Grudin, J. (2003). Personas: Practice and theory. In Proceedings of the 2003 Conference on Designing for User Experiences (pp. 1–15). ACM. doi
Quasi-experiment	A study that estimates the effect of an intervention without random assignment, using a comparison group or a time pattern chosen to rule out other explanations.	R13Experimentation and product analyticsFrom the field · not in the guide	What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0 (WWC 2022008). U.S. Department of Education, Institute of Education Sciences. ies.ed.gov Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
Question order effect	A change in how people answer a survey question caused by the questions that came before it.	R05Evaluative researchFrom the field · not in the guide	Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording, and context. Academic Press. archive.org Pew Research Center. (2021). Writing survey questions. pewresearch.org
R
R-squared	The proportion of the variation in an outcome that a regression model accounts for, from 0 to 1. A high value does not show that the model is causal.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge University Press. doi Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE.
Randomisation	Assigning units to groups by chance, so groups are balanced on measured and unmeasured factors.	R18Basic statistics for educational technologistsRelationship and cause · slide 39 (opens in a new tab)	Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books.
Randomisation unit	The thing chance assigns to a version: a student, a class, a teacher or a school.	R13Experimentation and product analyticsThe right unit · slide 24 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi Ritter, S., Murphy, A., Fancsali, S. E., Fitkariwala, V., Patel, N., & Lomas, J. D. (2020). UpGrade: An open source tool to support A/B testing in educational software [Paper presentation]. Workshop on Educational A/B Testing at Scale, Learning @ Scale 2020. upgradeplatform.org What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0 (WWC 2022008). U.S. Department of Education, Institute of Education Sciences. ies.ed.gov
Randomised controlled trial (RCT)	A study that assigns participants at random to an intervention or a control condition, so that a difference in outcomes can be attributed to the intervention.	R13Experimentation and product analyticsFrom the field · not in the guide	What Works Clearinghouse. (2022). What Works Clearinghouse procedures and standards handbook, version 5.0 (WWC 2022008). U.S. Department of Education, Institute of Education Sciences. ies.ed.gov Torgerson, D. J., & Torgerson, C. J. (2008). Designing randomised trials in health, education and the social sciences: An introduction. Palgrave Macmillan. doi
Ransomware	Malicious software that encrypts an organisation's files, and often steals a copy, then demands payment for their return. Schools and districts are frequent targets.	R17Privacy, security and ethicsFrom the field · not in the guide	National Institute of Standards and Technology. (2024). The NIST Cybersecurity Framework (CSF) 2.0 (NIST CSWP 29). National Institute of Standards and Technology. doi Cichonski, P., Millar, T., Grance, T., & Scarfone, K. (2012). Computer security incident handling guide (NIST Special Publication 800-61, Rev. 2). National Institute of Standards and Technology. doi
Rapid prototyping	Building and testing rough working versions early and repeatedly, so that design and evaluation run together instead of one after the other.	R07Design methods IIFrom the field · not in the guide	Tripp, S. D., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative instructional design strategy. Educational Technology Research and Development, 38(1), 31–44. doi
Rapid-cycle evaluation	A quick, low-cost study, often a comparison run over weeks with data a school already holds, of whether an education technology is working well enough to keep, change or drop.	R10Lean product developmentFrom the field · not in the guide	Mathematica. (2017, January 24). Providing timely and reliable evidence for schools at no cost [News release]. mathematica.org
Rapport	The working trust between interviewer and participant that makes a participant willing to speak frankly.	R04Discovery researchFrom the field · not in the guide	Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com Spradley, J. P. (1979). The ethnographic interview. Holt, Rinehart and Winston.
Re-identification	Working out who a record in a supposedly anonymous dataset belongs to, usually by linking it with other data. Small schools and rare characteristics make students easy to single out.	R17Privacy, security and ethicsFrom the field · not in the guide	Narayanan, A., & Shmatikov, V. (2008). Robust de-anonymization of large sparse datasets. In 2008 IEEE Symposium on Security and Privacy (pp. 111–125). IEEE. doi Sweeney, L. (2002). k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(5), 557–570. doi
Re-invention	Changes that users make to an innovation as they adopt it. Some re-invention helps a tool fit local conditions; too much removes what made it work.	R19Adoption and change in schoolsFrom the field · not in the guide	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press. Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk
Reaction	Feedback that reports how a design makes someone feel.	R07Design methods IICritique against the goal · slide 32 (opens in a new tab)	Connor, A. (2016). 3 kinds of feedback. Discussing Design, Medium. medium.com Connor, A., & Irizarry, A. (2015). Discussing design: Improving communication and collaboration through critique. O'Reilly Media. oreilly.com
Readability formula	A calculation that estimates how hard a text is to read from measures such as sentence length and word length; Flesch Reading Ease is the best known.	R20Communicating change to schoolsFrom the field · not in the guide	Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221–233. doi
Readiness for change	How far the people in an organisation are both willing and able to carry out a particular change. It is assessed before a rollout begins.	R19Adoption and change in schoolsFrom the field · not in the guide	Sharples, J., Eaton, J., & Boughelaf, J. (2024). A school's guide to implementation: Guidance report. Education Endowment Foundation. educationendowmentfoundation.org.uk Weiner, B. J. (2009). A theory of organizational readiness for change. Implementation Science, 4, Article 67. doi Armenakis, A. A., Harris, S. G., & Mossholder, K. W. (1993). Creating readiness for organizational change. Human Relations, 46(6), 681–703. doi
Reading age	The age at which a typical reader could understand a text, used as a target when writing for a broad audience.	R20Communicating change to schoolsFrom the field · not in the guide
Recall bias	Error that arises because people remember past events incompletely or inaccurately, and remember recent or striking events best.	R04Discovery researchFrom the field · not in the guide	Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. doi Coughlin, S. S. (1990). Recall bias in epidemiologic studies. Journal of Clinical Epidemiology, 43(1), 87–91. doi
Recognition over recall	Designing so users choose from visible options instead of retrieving information from memory.	R08UX foundations and design systemsLess to remember · slide 19 (opens in a new tab)	Budiu, R. (2024). Memory recognition and recall in user interfaces. Nielsen Norman Group. nngroup.com Nielsen, J. (2024). 10 usability heuristics for user interface design. Nielsen Norman Group. (Original work published 1994) nngroup.com
Record linkage	Matching records that belong to the same person across datasets, using a shared identifier or by comparing details such as name and date of birth.	R14Data strategy and governanceFrom the field · not in the guide	Government Accountability Office. (2014). Education and workforce data: Challenges in matching student and worker information raise concerns about longitudinal data systems (GAO-15-27). gao.gov Fellegi, I. P., & Sunter, A. B. (1969). A theory for record linkage. Journal of the American Statistical Association, 64(328), 1183–1210. doi
Recovery point objective (RPO)	The most data, measured as time before a failure, that an organisation accepts losing. An RPO of one hour requires backups or replication at least hourly.	R12DevOps and continuous deliveryFrom the field · not in the guide	Swanson, M., Bowen, P., Phillips, A. W., Gallup, D., & Lynes, D. (2010). Contingency planning guide for federal information systems (NIST Special Publication 800-34 Rev. 1). National Institute of Standards and Technology. doi
Recovery time objective (RTO)	The longest a service may stay unavailable after a major failure before the harm becomes unacceptable. It sets how fast recovery arrangements must work.	R12DevOps and continuous deliveryFrom the field · not in the guide	Swanson, M., Bowen, P., Phillips, A. W., Gallup, D., & Lynes, D. (2010). Contingency planning guide for federal information systems (NIST Special Publication 800-34 Rev. 1). National Institute of Standards and Technology. doi
Recurrence	How often a problem that was thought fixed comes back.	R22Support and feedback loopsTime to fix · slide 39 (opens in a new tab)	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com
Red-teaming	Deliberately trying to make a system fail or behave unsafely, to find problems before users do.	R16AI in learning productsEvaluate first · slide 19 (opens in a new tab)	National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. doi Department for Education. (2026). Generative AI: Product safety standards (Updated January 19, 2026). GOV.UK. gov.uk
Refactoring	Improving the structure of existing code without changing what it does; a common way to repay technical debt.	R11Software development for non-engineersThe spine · slide 13 (opens in a new tab)	Cunningham, W. (1992). The WyCash portfolio management system. In Addendum to the proceedings on object-oriented programming systems, languages, and applications (OOPSLA '92) (pp. 29–30). ACM. doi Fowler, M. (1999). Refactoring: Improving the design of existing code. Addison-Wesley.
Reference class forecasting	Forecasting from the outcomes of similar past projects, then adjusting for the case at hand (Flyvbjerg, 2006).	R09Agile and Scrum in practiceThe school calendar · slide 40 (opens in a new tab)	Flyvbjerg, B. (2006). From Nobel Prize to project management: Getting risks right. Project Management Journal, 37(3), 5–15. doi Kahneman, D., & Tversky, A. (1979). Intuitive prediction: Biases and corrective procedures. In S. Makridakis & S. C. Wheelwright (Eds.), Forecasting (TIMS Studies in the Management Sciences, Vol. 12, pp. 313–327). North-Holland.
Reference data	Agreed lists of codes and categories used to classify other data, such as school type, year group or ethnicity codes.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Reflection-in-action	Thinking about what one is doing while doing it, and adjusting the next move in response to what the situation shows.	R03Design thinking, used honestlyFrom the field · not in the guide	Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Basic Books.
Reflective practice	The habit of examining one's own work, during and after it, in order to learn from it and change what one does next.	R21Professional learning that changes practiceFrom the field · not in the guide	Schön, D. A. (1983). The reflective practitioner: How professionals think in action. Basic Books.
Reflexive loop	The way beliefs influence what data we select next time, so that an early reading of a few sessions decides what the researcher notices in the rest (Ross, 1994).	R04Discovery researchNotes and meanings · slide 37 (opens in a new tab)	Ross, R. (1994). The ladder of inference. In P. M. Senge, A. Kleiner, C. Roberts, R. B. Ross, & B. J. Smith, The fifth discipline fieldbook: Strategies and tools for building a learning organization. Currency Doubleday. archive.org
Reflexive thematic analysis	An approach to thematic analysis in which the researcher's interpretation is part of the method (Braun & Clarke, 2021).	R06Design methods ISynthesise together · slide 20 (opens in a new tab)	Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. doi Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. doi
Reflexivity	The analyst's examination of how their own position, assumptions and choices have shaped what they noticed in the data and what they made of it.	R06Design methods IFrom the field · not in the guide	Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. doi Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. doi
Reframing	Replacing the first description of a problem with another that opens different solutions. R06 Shifting semantic perspective in order to see things in a new way, one of the main ways insights are formed (Kolko, 2010).	R03Design thinking, used honestlyStay in the problem · slide 20 (opens in a new tab) R06Design methods IThe spine · slide 15 (opens in a new tab)	Dorst, K. (2011). The core of 'design thinking' and its application. Design Studies, 32(6), 521–532. doi Kolko, J. (2010). Abductive thinking and sensemaking: The drivers of design synthesis. Design Issues, 26(1), 15–28. doi Dorst, K. (2015). Frame innovation: Create new thinking by design. MIT Press. mitpress.mit.edu
Regression discontinuity	A design that compares students just above and just below a cutoff, such as a scholarship score, who are otherwise comparable. Only valid near the cutoff (Thistlethwaite & Campbell, 1960).	R18Basic statistics for educational technologistsRelationship and cause · slide 40 (opens in a new tab)	Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309–317. doi Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press.
Regression test	A test re-run after every change to confirm that features which used to work still do. A regression is a fault introduced into something that previously worked.	R11Software development for non-engineersFrom the field · not in the guide	Myers, G. J. (1979). The art of software testing. Wiley.
Regression to the mean	Extreme results are partly luck, and luck doesn't repeat, so schools chosen because they scored lowest will tend to score higher next time, with or without help.	R18Basic statistics for educational technologistsWhere numbers come from · slide 53 (opens in a new tab)	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Galton, F. (1886). Regression towards mediocrity in hereditary stature. The Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246–263. doi
Rehearsal	Practising a technique outside the classroom, usually with colleagues playing the students, so that it can be tried and corrected before a real lesson.	R21Professional learning that changes practiceFrom the field · not in the guide	Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov Lampert, M., Franke, M. L., Kazemi, E., Ghousseini, H., Turrou, A. C., Beasley, H., Cunard, A., & Crowe, K. (2013). Keeping it complex: Using rehearsals to support novice teacher learning of ambitious teaching. Journal of Teacher Education, 64(3), 226–243. doi
Reinforcement learning from human feedback (RLHF)	A training stage in which people rate or rank a model's outputs and the model is adjusted towards the responses they prefer. It is how chat assistants are made helpful and polite.	R16AI in learning productsFrom the field · not in the guide	Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744. doi
Reinforcing loop	A feedback loop in which a change feeds back to produce more change in the same direction, giving growth or collapse that speeds up.	R02Service design and systems thinkingFrom the field · not in the guide	Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green. Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. Doubleday/Currency.
Relative advantage	How much better an innovation is seen to be than what it replaces. It is the strongest of Rogers's five predictors of how fast an innovation is adopted.	R19Adoption and change in schoolsFrom the field · not in the guide	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Release	To make a change visible and available to users.	R12DevOps and continuous deliveryDeploy is not release · slide 30 (opens in a new tab)	Hodgson, P. (2017). Feature toggles (aka feature flags). martinfowler.com. martinfowler.com
Release candidate	A specific build judged potentially fit for release, which goes to users unchanged unless the remaining stages of testing find a reason to reject it.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Release note	An announcement of the changes in one release, drawn from the changelog and shaped for its readers.	R20Communicating change to schoolsWhat was removed · slide 40 (opens in a new tab)	Abebe, S. L., Ali, N., & Hassan, A. E. (2016). An empirical study of software release notes. Empirical Software Engineering, 21(3), 1107–1142. doi Wu, J., He, H., Gao, K., Xiao, W., Li, J., & Zhou, M. (2024). A comprehensive analysis of challenges and strategies for software release notes on GitHub. Empirical Software Engineering, 29, Article 104. doi Lacan, O. (2026). Keep a Changelog (Version 2.0.0). keepachangelog.com
Reliability	The consistency of a measure across items, occasions or raters.	R18Basic statistics for educational technologistsWhere numbers come from · slide 51 (opens in a new tab)	Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. doi Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. doi
Remote proctoring	Invigilating an examination through the candidate's own device, using webcam, microphone, screen recording and often automated flagging of suspect behaviour.	R17Privacy, security and ethicsFrom the field · not in the guide	Coghlan, S., Miller, T., & Paterson, J. (2021). Good proctor or "Big Brother"? Ethics of online exam supervision technologies. Philosophy & Technology, 34(4), 1581–1606. doi
Reopen rate	The share of closed tickets that users open again because the answer did not solve their problem.	R22Support and feedback loopsFrom the field · not in the guide
Replay enactment	A prototype test that replays real, recorded data at its original speed while users role-play a task (Holstein et al., 2019).	R07Design methods IIMatch the fidelity · slide 28 (opens in a new tab)	Holstein, K., McLaren, B. M., & Aleven, V. (2019). Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. Journal of Learning Analytics, 6(2), 27–52. doi Holstein, K., Harpstead, E., Gulotta, R., & Forlizzi, J. (2020). Replay enactments: Exploring possible futures through historical data. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. ACM. doi
Request for proposal (RFP)	A document a buyer issues to invite suppliers to bid for a piece of work, setting out what is needed and how bids will be judged.	R01Digital product managementFrom the field · not in the guide	Hopson, M., McFadden, V., Refoy, R., & Rouault, A. (Eds.). (2020). De-risking government technology: Federal agency field guide. 18F, U.S. General Services Administration. guides.18f.gov Morrison, J. R., Ross, S. M., & Corcoran, R. P. (2014). Fostering market efficiency in K-12 ed-tech procurement. Johns Hopkins University Center for Research and Reform in Education; Digital Promise. digitalpromise.org
Research repository	A central store of research outputs that others in an organisation can search (Rosala, 2024a).	R06Design methods IKeep the trail · slide 28 (opens in a new tab)	Rosala, M. (2024a). Research repositories for tracking UX research and growing your ResearchOps. Nielsen Norman Group. nngroup.com Rosala, M. (2024b). Why research repositories fail and how to get them right. Nielsen Norman Group. nngroup.com
Researcher degrees of freedom	The many choices in collecting and analysing data that can push a result towards significance (Simmons et al., 2011).	R13Experimentation and product analyticsWrite it down first · slide 32 (opens in a new tab)	Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460–465. doi
ResearchOps	The people, processes and tools that support researchers and let research scale, covering recruitment, consent, tooling, data handling and the sharing of findings.	R06Design methods IFrom the field · not in the guide	Rosala, M. (2024a). Research repositories for tracking UX research and growing your ResearchOps. Nielsen Norman Group. nngroup.com
Resistance to change	Reluctance or refusal to take up a new way of working. It is often a reasonable response to extra effort, unclear benefit or a poor past experience.	R19Adoption and change in schoolsFrom the field · not in the guide	Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47(4), 47–61. doi Coch, L., & French, J. R. P., Jr. (1948). Overcoming resistance to change. Human Relations, 1(4), 512–532. doi
Response order effect	A change in which answer option people choose caused by the order in which the options are presented, such as favouring the first option in a written list.	R05Evaluative researchFrom the field · not in the guide	Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. doi Pew Research Center. (2021). Writing survey questions. pewresearch.org
Response rate	The share of the people invited to take a survey who complete it. A low rate is a warning of possible nonresponse bias, not proof of it.	R05Evaluative researchFrom the field · not in the guide	Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167–189. doi Davern, M. (2013). Nonresponse rates are a problematic indicator of nonresponse bias in survey research. Health Services Research, 48(3), 905–912. doi
Responsive design	Building one set of pages whose layout adapts to the width of the screen, using flexible grids, flexible images and media queries.	R08UX foundations and design systemsFrom the field · not in the guide	Marcotte, E. (2011). Responsive web design. A Book Apart.
REST	Representational state transfer: the architectural style of the web, in which clients act on named resources through a small uniform set of requests. Most web APIs today are described as RESTful.	R11Software development for non-engineersFrom the field · not in the guide	Fielding, R. T. (2000). Architectural styles and the design of network-based software architectures [Doctoral dissertation, University of California, Irvine]. Fielding, R. T., & Taylor, R. N. (2002). Principled design of the modern Web architecture. ACM Transactions on Internet Technology, 2(2), 115–150. doi
Retention schedule	A list of datasets with how long each is kept, why, and who deletes it.	R14Data strategy and governanceThe deletion date · slide 44 (opens in a new tab)	Information and Records Management Society. (2019). Academies toolkit: Pupil records guidance. irms.org.uk Privacy Technical Assistance Center. (2014). Best practices for data destruction (PTAC-IB-5). U.S. Department of Education. files.eric.ed.gov European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. eur-lex.europa.eu
Retrieval	Finding passages in an approved library and passing them to a model with the question, a common way to ground answers.	R16AI in learning productsGround and check · slide 46 (opens in a new tab)	Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33.
Retrieval-augmented generation (RAG)	A design in which the system first searches a chosen collection of documents and then passes the passages it finds to the model to answer from.	R16AI in learning productsFrom the field · not in the guide	Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474.
Retrospective	A regular meeting at which a team inspects its own way of working and decides what to change.	R09Agile and Scrum in practiceInspect the process · slide 33 (opens in a new tab)	Derby, E., & Larsen, D. (2006). Agile retrospectives: Making good teams great. Pragmatic Bookshelf. pragprog.com Kerth, N. L. (2001). Project retrospectives: A handbook for team reviews. Dorset House. Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
RICE scoring	A prioritisation score of reach, impact and confidence multiplied together and divided by effort, useful for comparing many small items (McBride, 2018).	R01Digital product managementSay no in public · slide 31 (opens in a new tab)	McBride, S. (2018, January 5). RICE prioritization framework for product managers. Intercom. intercom.com
Rich picture	A free-hand drawing of a problem situation showing its people, relationships, conflicts and concerns, made to share understanding before any formal model is drawn.	R02Service design and systems thinkingFrom the field · not in the guide	Checkland, P., & Scholes, J. (1990). Soft systems methodology in action. Wiley. doi
Right to be heard	A child's right to express views freely in all matters affecting them, with those views given due weight in accordance with the child's age and maturity (United Nations, 1989).	R04Discovery researchTwo yeses · slide 44 (opens in a new tab)	United Nations. (1989). Convention on the Rights of the Child (General Assembly resolution 44/25). ohchr.org Lundy, L. (2007). ‘Voice’ is not enough: Conceptualising Article 12 of the United Nations Convention on the Rights of the Child. British Educational Research Journal, 33(6), 927–942. doi British Educational Research Association. (2024). Ethical guidelines for educational research (5th ed.). bera.ac.uk
Rising cost of defects	The popular claim that a defect costs up to 100 times more to fix after release; the original data were thin, and 171 projects showed no consistent effect (Menzies et al., 2017).	R11Software development for non-engineersThe cost of change · slide 20 (opens in a new tab)	Menzies, T., Nichols, W., Shull, F., & Layman, L. (2017). Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle. Empirical Software Engineering, 22(4), 1903–1935. doi Bossavit, L. (2015). The leprechauns of software engineering: How folklore turns into fact and what to do about it. Leanpub. leanpub.com
Riskiest assumption	The belief that is most important to the plan and least supported by evidence.	R10Lean product developmentThe riskiest assumption · slide 18 (opens in a new tab)	Bland, D. J., & Osterwalder, A. (2019). Testing business ideas: A field guide for rapid experimentation. Wiley. books.google.com Bland, D. J. (2020, August 4). How assumptions mapping can focus your teams on running experiments that matter. Strategyzer. strategyzer.com Government Digital Service. (2019). How the alpha phase works. GOV.UK Service Manual. gov.uk
Roadmap	A statement of intent and direction, ideally ordered by certainty rather than by date.	R01Digital product managementRoadmap as hypothesis · slide 24 (opens in a new tab)	Lombardo, C. T., McCarthy, B., Ryan, E., & Connors, M. (2017). Product roadmaps relaunched: How to set direction while embracing uncertainty. O'Reilly Media.
Rollback	Returning a service to its previous version after a faulty release.	R12DevOps and continuous deliverySmall and reversible · slide 26 (opens in a new tab)	Fowler, M. (2010). Blue green deployment. martinfowler.com. martinfowler.com Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Root cause analysis	A structured investigation that traces a fault back to the underlying conditions that produced it, so that the fix prevents it from returning.	R22Support and feedback loopsFrom the field · not in the guide
Runbook	Written step-by-step instructions for a routine operational task or for responding to a particular alert, so that whoever is on call can act without working it out afresh.	R12DevOps and continuous deliveryFrom the field · not in the guide	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google Limoncelli, T. A., Chalup, S. R., & Hogan, C. J. (2014). The practice of cloud system administration: Designing and operating large distributed systems (Vol. 2). Addison-Wesley.
S
Sample	The subset of a population actually observed.	R18Basic statistics for educational technologistsWhat statistics is for · slide 8 (opens in a new tab)	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican.
Sample ratio mismatch (SRM)	A split of users between groups that differs from the planned ratio by more than chance allows. It signals a fault in assignment or logging, so the results should not be trusted.	R13Experimentation and product analyticsFrom the field · not in the guide	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., & Dmitriev, P. (2019). Diagnosing sample ratio mismatch in online controlled experiments: A taxonomy and rules of thumb for practitioners. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2156–2164). ACM. doi
Sampling frame	The list from which a survey sample is drawn, such as a register of teachers. People missing from the list cannot be selected, whatever the sample size.	R05Evaluative researchFrom the field · not in the guide	Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.
SAMR model	Puentedura's four levels of classroom technology use: substitution, augmentation, modification and redefinition. It is widely used in schools, though its research basis is thin.	R19Adoption and change in schoolsFrom the field · not in the guide	Hamilton, E. R., Rosenberg, J. M., & Akcaoglu, M. (2016). The Substitution Augmentation Modification Redefinition (SAMR) model: A critical review and suggestions for its use. TechTrends, 60(5), 433–441. doi
Satisficing	Giving an answer that is good enough rather than accurate, to save effort (Krosnick, 1991).	R05Evaluative researchQuestions that cannot lead · slide 32 (opens in a new tab)	Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. doi Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Review, 63(2), 129–138. doi
Saturation	The point at which enough data has been collected; one study of 60 interviews found it within the first twelve, which is not a general rule (Guest et al., 2006).	R06Design methods ISynthesise together · slide 20 (opens in a new tab)	Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. doi
Say–do gap	The difference between what people report about their behaviour and what they actually do.	R04Discovery researchThe misconception · slide 9 (opens in a new tab)	LaPiere, R. T. (1934). Attitudes vs. actions. Social Forces, 13(2), 230–237. doi Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231–259. doi Nielsen, J. (2001). First rule of usability? Don't listen to users. Nielsen Norman Group. nngroup.com
Scalability	A system's ability to handle more users, data or requests by adding resources, without being redesigned. Scaling up uses a bigger machine; scaling out uses more machines.	R11Software development for non-engineersFrom the field · not in the guide	Kleppmann, M. (2017). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. O'Reilly Media.
Scale-up	Extending a tool or practice from a few sites to many. In Coburn's account, real scale means depth, sustainability, spread and a shift in ownership, not only numbers.	R19Adoption and change in schoolsFrom the field · not in the guide	Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017). Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. Journal of Medical Internet Research, 19(11), Article e367. doi Coburn, C. E. (2003). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3–12. doi
Scaled Agile Framework (SAFe)	A scaling framework that promises a common process across many teams, based on ten Lean-Agile principles, and criticised as a return to heavy, prescriptive process (Scaled Agile, n.d.).	R09Agile and Scrum in practiceInspect the process · slide 34 (opens in a new tab)	Scaled Agile. (n.d.). SAFe Lean-Agile principles. Retrieved September 17, 2026, from framework.scaledagile.com Schwaber, K. (2013, August 6). unSAFe at any speed. Ken Schwaber's Blog. kenschwaber.wordpress.com Digital.ai. (2023). The 17th State of Agile report. digital.ai
Scenario	A short written story of a particular person using a product to reach a goal in a particular setting, used to think through a design before building it.	R07Design methods IIFrom the field · not in the guide	Carroll, J. M. (2000). Making use: Scenario-based design of human-computer interactions. MIT Press. doi
School official exception	The FERPA provision that lets a school share education records with a vendor without parental consent, if the vendor does a job the school would otherwise do, under the school's direct control.	R17Privacy, security and ethicsFrom the field · not in the guide	Family Educational Rights and Privacy, 34 C.F.R. Part 99.
Scope creep	The gradual, unplanned growth of what a piece of work is expected to deliver, without matching changes to time, budget or people.	R01Digital product managementFrom the field · not in the guide	Project Management Institute. (2021). A guide to the project management body of knowledge (PMBOK guide) (7th ed.).
Screen reader	Software that reads out the content and controls of a screen as speech, or sends them to a braille display, for people who are blind or have low vision.	R08UX foundations and design systemsFrom the field · not in the guide	World Wide Web Consortium. (2023). Web Content Accessibility Guidelines (WCAG) 2.2 (W3C Recommendation). w3.org Horton, S., & Quesenbery, W. (2013). A web for everyone: Designing accessible user experiences. Rosenfeld Media.
Screener	A short set of questions put to possible participants before a study, used to select those who fit the recruitment criteria and leave out those who do not.	R04Discovery researchFrom the field · not in the guide	Government Digital Service. (2020). Find user research participants. GOV.UK Service Manual. gov.uk Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com
Scrum	A framework founded on empiricism and lean thinking, with three accountabilities, five events and three artefacts, whose events create points where a team can inspect and adapt (Schwaber & Sutherland, 2020).	R09Agile and Scrum in practiceThe spine · slide 16 (opens in a new tab)	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org Takeuchi, H., & Nonaka, I. (1986). The new new product development game. Harvard Business Review, 64(1), 137–146.
Scrum Master	The person in a Scrum team accountable for how well the team uses Scrum, who coaches the team and helps remove whatever is blocking its progress.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Scrum of Scrums	A short, regular meeting of representatives from several Scrum teams working on the same product, held to coordinate their work and resolve dependencies between them.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K. (2004). Agile project management with Scrum. Microsoft Press. doi
Second-order barrier	An internal obstacle, such as beliefs about teaching or confidence (Ertmer, 1999).	R19Adoption and change in schoolsLeader, then department · slide 33 (opens in a new tab)	Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47(4), 47–61. doi Ertmer, P. A. (2005). Teacher pedagogical beliefs: The final frontier in our quest for technology integration? Educational Technology Research and Development, 53(4), 25–39. doi
Seductive details	Interesting and irrelevant information added to a lesson; learners given it tended to perform worse than those who learned without it (Sundararajan & Adesope, 2020).	R08UX foundations and design systemsThe misconception · slide 9 (opens in a new tab)	Sundararajan, N., & Adesope, O. (2020). Keep it coherent: A meta-analysis of the seductive details effect. Educational Psychology Review, 32(3), 707–734. doi Mayer, R. E. (2024). The past, present, and future of the cognitive theory of multimedia learning. Educational Psychology Review, 36(1), Article 8. doi
Selection bias	Distortion that arises when the people in a sample or a comparison group differ systematically from those they are meant to represent, such as schools that volunteered for a pilot.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Angrist, J. D., & Pischke, J.-S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton University Press. Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161. doi
Self-regulated learning (SRL)	Learning in which students set goals, plan, monitor their progress and adjust their approach. Much learning analytics research tries to measure or support it.	R15Learning analyticsFrom the field · not in the guide	Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. doi Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64–70. doi
Self-service	Support that users get for themselves, through help articles, guided fixes or account tools, without contacting a person.	R22Support and feedback loopsFrom the field · not in the guide	Government Digital Service. (2016). Set up and manage user support. GOV.UK Service Manual. gov.uk
Semantic and latent coding	Semantic coding records what participants said, at face value. Latent coding records the assumptions and ideas that lie beneath what was said.	R06Design methods IFrom the field · not in the guide	Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. doi
Semantic versioning	A version numbering scheme of major, minor and patch numbers that signals whether a release breaks existing use (Preston-Werner, n.d.).	R08UX foundations and design systemsRun it as a product · slide 44 (opens in a new tab)	Preston-Werner, T. (n.d.). Semantic versioning 2.0.0. Retrieved September 17, 2026, from semver.org Government Digital Service. (n.d.-c). Versioning [GOV.UK Frontend documentation]. GitHub. Retrieved September 17, 2026, from github.com
Semi-structured interview	An interview that follows a prepared list of topics but lets the interviewer change the order and follow up on whatever the participant raises.	R04Discovery researchFrom the field · not in the guide	Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com Brinkmann, S., & Kvale, S. (2015). InterViews: Learning the craft of qualitative research interviewing (3rd ed.). SAGE.
Sense of urgency	A shared belief that a change is needed now. Kotter makes establishing it the first step of a change effort, before any solution is announced.	R20Communicating change to schoolsFrom the field · not in the guide	Kotter, J. P. (1995). Leading change: Why transformation efforts fail. Harvard Business Review, 73(2), 59–67. hbr.org Kotter, J. P. (1996). Leading change. Harvard Business School Press. doi
Sensegiving	A leader's attempt to shape how others interpret a change, by offering a preferred account of what it means and why it matters.	R20Communicating change to schoolsFrom the field · not in the guide	Spillane, J. P., Reiser, B. J., & Reimer, T. (2002). Policy implementation and cognition: Reframing and refocusing implementation research. Review of Educational Research, 72(3), 387–431. doi Gioia, D. A., & Chittipeddi, K. (1991). Sensemaking and sensegiving in strategic change initiation. Strategic Management Journal, 12(6), 433–448. doi
Sensemaking	The process by which people interpret a change in light of what they already know and do (Spillane et al., 2002).	R20Communicating change to schoolsThe spine · slide 14 (opens in a new tab)	Spillane, J. P., Reiser, B. J., & Reimer, T. (2002). Policy implementation and cognition: Reframing and refocusing implementation research. Review of Educational Research, 72(3), 387–431. doi Coburn, C. E. (2001). Collective sensemaking about reading: How teachers mediate reading policy in their professional communities. Educational Evaluation and Policy Analysis, 23(2), 145–170. doi Coburn, C. E. (2005). Shaping teacher sensemaking: School leaders and the enactment of reading policy. Educational Policy, 19(3), 476–509. doi
Sensitivity	Of the students who had the outcome, the share the flag caught.	R15Learning analyticsThe base rate · slide 40 (opens in a new tab)	Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77–100. doi Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. doi
Sensitivity and specificity	Sensitivity is the share of students who need help that a tool catches; specificity is the share of those who don't that it clears. No threshold removes the trade-off between them.	R18Basic statistics for educational technologistsRisk and base rates · slide 45 (opens in a new tab)	Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M., & Woloshin, S. (2007). Helping doctors and patients make sense of health statistics. Psychological Science in the Public Interest, 8(2), 53–96. doi Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.
Sequential testing	Methods that allow a running experiment to be analysed at planned or repeated points and stopped early, while keeping the false positive rate at its stated level.	R13Experimentation and product analyticsFrom the field · not in the guide	Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2022). Always valid inference: Continuous monitoring of A/B tests. Operations Research, 70(3), 1806–1821. doi Wald, A. (1947). Sequential analysis. John Wiley & Sons. doi
Serial reproduction	Experiments in which material is retold from one person to the next; they suggest familiar material survives retelling better than unfamiliar material (Ost et al., 2022).	R20Communicating change to schoolsThe spine · slide 14 (opens in a new tab)	Ost, J., Udell, J., Dear, S., Zinken, J., Blank, H., & Costall, A. (2022). The serial reproduction of an urban myth: Revisiting Bartlett's schema theory. Memory, 30(6), 775–783. doi Bartlett, F. C. (1932). Remembering: A study in experimental and social psychology. Cambridge University Press. doi
Service	Everything that has to happen for a user to get what they came for, across people, rules and systems.	R02Service design and systems thinkingThe misconception · slide 10 (opens in a new tab)	Downe, L. (2020). Good services: How to design services that work. BIS Publishers. good.services Government Digital Service. (2019). Service Standard. GOV.UK. gov.uk Polaine, A., Løvlie, L., & Reason, B. (2013). Service design: From insight to implementation. Rosenfeld Media.
Service blueprint	A diagram that lines up user actions with the frontstage, backstage and supporting work behind them.	R02Service design and systems thinkingMap the backstage · slide 19 (opens in a new tab)	Shostack, G. L. (1984). Designing services that deliver. Harvard Business Review, 62(1), 133–139. hbr.org Bitner, M. J., Ostrom, A. L., & Morgan, F. N. (2008). Service blueprinting: A practical technique for service innovation. California Management Review, 50(3), 66–94. doi Gibbons, S. (2017, August 27). Service blueprints: Definition. Nielsen Norman Group. nngroup.com
Service catalogue	The published list of services a provider offers, with what each includes, who may ask for it and how.	R22Support and feedback loopsFrom the field · not in the guide	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com
Service design	The work of making the whole service visible, its people, rules, handovers and other systems as well as screens, and of treating rules, forms and deadlines as design material.	R02Service design and systems thinkingThe spine · slide 12 (opens in a new tab)	Polaine, A., Løvlie, L., & Reason, B. (2013). Service design: From insight to implementation. Rosenfeld Media. Stickdorn, M., Hormess, M. E., Lawrence, A., & Schneider, J. (2018). This is service design doing: Applying service design thinking in the real world. O'Reilly Media. Downe, L. (2020). Good services: How to design services that work. BIS Publishers. good.services
Service desk	The single point of contact between a service provider and its users, which receives incidents and requests and sees them through to resolution.	R22Support and feedback loopsFrom the field · not in the guide	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com
Service level	A target for how quickly a type of request will be answered or resolved.	R22Support and feedback loopsTime to fix · slide 40 (opens in a new tab)	Government Digital Service. (2016). Set up and manage user support. GOV.UK Service Manual. gov.uk Department for Education. (2026). IT support standards for schools and colleges. In Meeting digital and technology standards in schools and colleges. GOV.UK. gov.uk
Service level agreement (SLA)	A documented agreement between a service provider and a customer that states the services to be provided and the levels of service expected, such as response and resolution times.	R22Support and feedback loopsFrom the field · not in the guide	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com International Organization for Standardization. (2018). Information technology — Service management — Part 1: Service management system requirements (ISO/IEC 20000-1:2018).
Service level indicator (SLI)	A defined measure of some aspect of a service, such as the share of logins that succeed.	R11Software development for non-engineersKnowing it is broken · slide 38 (opens in a new tab)	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Service level objective (SLO)	A target value for a service level indicator, such as 99.9% of logins succeeding each month.	R11Software development for non-engineersKnowing it is broken · slide 38 (opens in a new tab)	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Service recovery	What an organisation does to put things right after it has failed a user: acknowledging the failure, fixing it, and making up for the trouble caused.	R22Support and feedback loopsFrom the field · not in the guide	Hart, C. W. L., Heskett, J. L., & Sasser, W. E., Jr. (1990). The profitable art of service recovery. Harvard Business Review, 68(4), 148–156.
Service request	A user's request for something that is a normal part of the service, such as a password reset or a new account, as distinct from a report of a fault.	R22Support and feedback loopsFrom the field · not in the guide	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com
Service safari	A research exercise in which team members use a service themselves, as its customers would, and record what they meet along the way.	R02Service design and systems thinkingFrom the field · not in the guide	Stickdorn, M., & Schneider, J. (Eds.). (2010). This is service design thinking: Basics, tools, cases. BIS Publishers. Stickdorn, M., Hormess, M. E., Lawrence, A., & Schneider, J. (2018). This is service design doing: Applying service design thinking in the real world. O'Reilly Media.
Service-dominant logic	The view that all economic exchange is an exchange of service, that goods are only a means of delivering it, and that value is created jointly with the user in use.	R02Service design and systems thinkingFrom the field · not in the guide	Vargo, S. L., & Lusch, R. F. (2004). Evolving to a new dominant logic for marketing. Journal of Marketing, 68(1), 1–17. doi
Servicescape	The physical setting in which a service is delivered, including its layout, signs and atmosphere, considered for its effect on users and staff.	R02Service design and systems thinkingFrom the field · not in the guide	Bitner, M. J. (1992). Servicescapes: The impact of physical surroundings on customers and employees. Journal of Marketing, 56(2), 57–71. doi
Set-based design	Developing several design options in parallel and eliminating them gradually as evidence arrives, instead of choosing one early and reworking it. Also called set-based concurrent engineering.	R10Lean product developmentFrom the field · not in the guide	Poppendieck, M., & Poppendieck, T. (2003). Lean software development: An agile toolkit. Addison-Wesley. informit.com Ward, A. C. (2007). Lean product and process development. Lean Enterprise Institute. Morgan, J. M., & Liker, J. K. (2006). The Toyota product development system: Integrating people, process, and technology. Productivity Press.
Severity level	A grade given to a fault according to how many users it affects and how badly, which sets how fast it must be handled.	R22Support and feedback loopsFrom the field · not in the guide
Severity rating	A grade given to each usability problem, usually based on how many users it affects and how badly, used to decide which problems to fix first.	R05Evaluative researchFrom the field · not in the guide	Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann. shop.elsevier.com Dumas, J. S., & Redish, J. C. (1999). A practical guide to usability testing (Rev. ed.). Intellect.
Shadowing	Following one person through their working day and recording what they do, where, with whom and with what tools.	R04Discovery researchFrom the field · not in the guide	McDonald, S. (2005). Studying actions in context: A qualitative shadowing method for organizational research. Qualitative Research, 5(4), 455–473. doi
Shelfware	Software that has been bought and licensed but is never or barely used.	R19Adoption and change in schoolsFrom the field · not in the guide	Glimpse K12. (2019, May 15). Glimpse K12 analysis of school spending shows that two-thirds of software license purchases go unused [Press release]. GlobeNewswire. globenewswire.com Wood, C. (2018). Most educational software licenses go unused in K-12 districts. EdScoop. edscoop.com
Shifting the burden	A pattern in which easy fixes relieve symptoms and weaken the fundamental solution (Senge, 1990).	R02Service design and systems thinkingPolicy as material · slide 37 (opens in a new tab)	Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. Doubleday/Currency. Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green.
Signifier	Any perceivable indicator that communicates appropriate behaviour to a person (Norman, 2013).	R08UX foundations and design systemsThe next step · slide 25 (opens in a new tab)	Norman, D. (2013). The design of everyday things (Rev. and expanded ed.). Basic Books. basicbooks.com Norman, D. A. (2008). Signifiers, not affordances. Interactions, 15(6), 18–19.
Silent failure	A fault that users meet and work around without reporting it.	R22Support and feedback loopsThe misconception · slide 10 (opens in a new tab)	Goodman, J. (1999). Basic facts on customer complaint behavior and the impact of service on the bottom line. Competitive Advantage, June, 1–5. newtoncomputing.com Hirschman, A. O. (1970). Exit, voice, and loyalty: Responses to decline in firms, organizations, and states. Harvard University Press. hup.harvard.edu
Simpson's paradox	A pattern that appears in every subgroup but reverses or vanishes when the subgroups are combined, because the groups differ in size or make-up.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Charig, C. R., Webb, D. R., Payne, S. R., & Wickham, J. E. A. (1986). Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. BMJ, 292(6524), 879–882. doi Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. Basic Books. Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2), 238–241. doi
Single Ease Question (SEQ)	A one-item questionnaire given straight after a task, asking the user to rate how difficult or easy it was on a seven-point scale.	R05Evaluative researchFrom the field · not in the guide	Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann. shop.elsevier.com Sauro, J., & Dumas, J. S. (2009). Comparison of three one-question, post-task usability questionnaires. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1599–1608). ACM. doi
Single point of failure	A part of a system with no backup, whose failure stops the whole service. Reliable designs remove them by duplicating components.	R11Software development for non-engineersFrom the field · not in the guide	Kleppmann, M. (2017). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. O'Reilly Media.
Single sign-on (SSO)	One login, usually the school's or ministry's identity service, that opens many applications, so that accounts are created, checked and closed in one place.	R17Privacy, security and ethicsFrom the field · not in the guide	Hardt, D. (Ed.). (2012). The OAuth 2.0 authorization framework (RFC 6749). Internet Engineering Task Force. doi Grassi, P. A., Garcia, M. E., & Fenton, J. L. (2017). Digital identity guidelines (NIST Special Publication 800-63-3). National Institute of Standards and Technology. doi
Site reliability engineering (SRE)	Google's approach to operations, which treats running a service as a software engineering problem: reliability targets are explicit, releases are governed by error budgets and repetitive work is automated away.	R12DevOps and continuous deliveryFrom the field · not in the guide	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Sketch	A quick, cheap, disposable representation made to explore rather than to confirm (Buxton, 2007). R07 A quick, cheap, disposable drawing made to explore an idea rather than confirm it (Buxton, 2007).	R03Design thinking, used honestlyPrototype to learn · slide 27 (opens in a new tab) R07Design methods IISketch many · slide 21 (opens in a new tab)	Buxton, B. (2007). Sketching user experiences: Getting the design right and the right design. Morgan Kaufmann. books.google.com Hartmann, B. (2009). Gaining design insight through interaction prototyping tools [Doctoral dissertation, Stanford University]. people.eecs.berkeley.edu Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006). Getting the right design and the design right: Testing many is better than one. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1243–1252). ACM. doi
Skew	Lopsidedness in a distribution, with a long tail on one side. In skewed data such as time spent on a platform, the mean and the median can differ widely.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican.
Slips and mistakes	Two kinds of user error. A slip is the right intention carried out wrongly, such as a mis-tap; a mistake is the wrong intention, formed from a faulty understanding.	R08UX foundations and design systemsFrom the field · not in the guide	Norman, D. (2013). The design of everyday things (Rev. and expanded ed.). Basic Books. basicbooks.com Reason, J. (1990). Human error. Cambridge University Press. doi
Sludge	Friction in a process, such as needless forms, waiting or repeated proof, that makes it harder for people to get something they want or are entitled to.	R02Service design and systems thinkingFrom the field · not in the guide	Herd, P., & Moynihan, D. P. (2018). Administrative burden: Policymaking by other means. Russell Sage Foundation. doi Sunstein, C. R. (2021). Sludge: What stops us from getting things done and what to do about it. MIT Press. doi
Small batches	Work broken into pieces that take hours to a couple of days, which shortens the time to feedback and makes problems easier to find and fix (DORA, 2025).	R12DevOps and continuous deliverySmall and reversible · slide 25 (opens in a new tab)	DORA. (2025). Capabilities: Working in small batches. Google Cloud. dora.dev Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com
Smoke test	A short set of checks run straight after a deployment to confirm the system starts and its most important functions respond, before anything more thorough is tried.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Snowball sampling	Recruiting by asking each participant to suggest others. It reaches groups that are hard to find, but tends to return people who resemble those already recruited.	R04Discovery researchFrom the field · not in the guide	Goodman, L. A. (1961). Snowball sampling. The Annals of Mathematical Statistics, 32(1), 148–170. doi
SOC 2	An independent auditor's report on a service provider's controls for security, availability, processing integrity, confidentiality and privacy. Type 1 covers design at a date; Type 2 covers operation over a period.	R17Privacy, security and ethicsFrom the field · not in the guide	American Institute of Certified Public Accountants. (2017). Trust services criteria for security, availability, processing integrity, confidentiality, and privacy. AICPA.
Social desirability	The pull towards answers that sound right or please the person asking.	R04Discovery researchThe misconception · slide 10 (opens in a new tab)	Nielsen, J. (2001). First rule of usability? Don't listen to users. Nielsen Norman Group. nngroup.com Crowne, D. P., & Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24(4), 349–354. doi
Social network analysis (SNA)	The study of patterns of connection between people. In learning analytics it maps who interacts with whom, for example in discussion forums.	R15Learning analyticsFrom the field · not in the guide	Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge University Press. doi
Sociotechnical system	A system in which people, their working arrangements and their technology depend on one another, so that changing the technology alone changes the work in ways nobody planned.	R02Service design and systems thinkingFrom the field · not in the guide	Norman, D. A., & Stappers, P. J. (2015). DesignX: Complex sociotechnical systems. She Ji: The Journal of Design, Economics, and Innovation, 1(2), 83–106. doi Trist, E. L., & Bamforth, K. W. (1951). Some social and psychological consequences of the longwall method of coal-getting. Human Relations, 4(1), 3–38. doi
Soft systems methodology	An approach to messy organisational problems in which the people involved build and compare models of the purposeful activity each of them sees, to agree changes all can accept.	R02Service design and systems thinkingFrom the field · not in the guide	Checkland, P. (1981). Systems thinking, systems practice. Wiley.
Software architecture	The main parts of a system, how they connect, and the early decisions about them that are costly to change later. It largely decides qualities such as speed, reliability and ease of change.	R11Software development for non-engineersFrom the field · not in the guide	ISO/IEC. (2023). Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model (ISO/IEC Standard No. 25010:2023). International Organization for Standardization. iso.org Bass, L., Clements, P., & Kazman, R. (1998). Software architecture in practice. Addison-Wesley.
Software as a service (SaaS)	Software that the supplier runs on its own cloud infrastructure and customers use over the internet, usually by subscription, with nothing to install or upgrade themselves.	R11Software development for non-engineersFrom the field · not in the guide	Mell, P., & Grance, T. (2011). The NIST definition of cloud computing (NIST Special Publication 800-145). National Institute of Standards and Technology. doi
Solutionism	The habit of recasting complex social situations as neat problems with technical fixes (Morozov, 2013).	R01Digital product managementWork back from the mandate · slide 39 (opens in a new tab)	Morozov, E. (2013). To save everything, click here: The folly of technological solutionism. PublicAffairs.
Spacing	Spreading learning across separate sessions rather than massing it in one (Cepeda et al., 2006).	R21Professional learning that changes practiceSpread it out · slide 26 (opens in a new tab)	Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380. doi Sims, S., Fletcher-Wood, H., O'Mara-Eves, A., Cottingham, S., Stansfield, C., Van Herwegen, J., & Anders, J. (2021). What are the characteristics of effective teacher professional development? A systematic review and meta-analysis. Education Endowment Foundation. files.eric.ed.gov
Spatial contiguity principle	People learn better when words and pictures are presented near rather than far from each other on the page or screen (Mayer, 2024).	R08UX foundations and design systemsThe misconception · slide 9 (opens in a new tab)	Mayer, R. E. (2024). The past, present, and future of the cognitive theory of multimedia learning. Educational Psychology Review, 36(1), Article 8. doi Mayer, R. E. (2021). Multimedia learning (3rd ed.). Cambridge University Press. doi
Special category data	Kinds of personal data given extra legal protection because misuse does serious harm, such as health, ethnicity, religion and biometric data. Special educational needs records usually fall here.	R17Privacy, security and ethicsFrom the field · not in the guide	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88.
Speculative design	Design work that makes objects and scenarios from possible futures, not for sale or use but to prompt debate about which futures people want.	R03Design thinking, used honestlyFrom the field · not in the guide	Dunne, A., & Raby, F. (2013). Speculative everything: Design, fiction, and social dreaming. MIT Press.
Spike	A short, timeboxed piece of work done to answer a technical or design question so that a later story can be estimated or built with less risk.	R09Agile and Scrum in practiceFrom the field · not in the guide	Cohn, M. (2004). User stories applied: For agile software development. Addison-Wesley. mountaingoatsoftware.com Beck, K., & Andres, C. (2004). Extreme programming explained: Embrace change (2nd ed.). Addison-Wesley.
Sprint	A fixed period of one month or less in which a Scrum team works towards a Sprint Goal.	R09Agile and Scrum in practiceThe spine · slide 16 (opens in a new tab)	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org Schwaber, K., & Beedle, M. (2002). Agile software development with Scrum. Prentice Hall. doi
Sprint Goal	The single objective a Scrum team commits to for a sprint, which gives the chosen backlog items a shared purpose.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Sprint Planning	The event that opens a sprint, at which the Scrum team agrees why the sprint matters, which backlog items it will take on and how the work will be done.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Sprint Review	The Scrum event at which the team and stakeholders inspect the outcome of a sprint and decide what to adapt.	R09Agile and Scrum in practiceInspect the process · slide 35 (opens in a new tab)	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org Government Digital Service. (2016, May 23). Governance principles for agile service delivery. GOV.UK Service Manual. gov.uk
Stage-gate process	A way of running product development as a fixed series of stages, each ending in a review at which managers approve, stop or send back the work.	R01Digital product managementFrom the field · not in the guide	Cooper, R. G. (1990). Stage-gate systems: A new tool for managing new products. Business Horizons, 33(3), 44–54. doi
Staged rollout	Opening a deployed change by feature flag to staff, then pilot schools, then half of schools, then everyone, and switching the flag off if a signal goes wrong.	R12DevOps and continuous deliveryDeploy is not release · slide 31 (opens in a new tab)	Hodgson, P. (2017). Feature toggles (aka feature flags). martinfowler.com. martinfowler.com Sato, D. (2014). Canary release. martinfowler.com. martinfowler.com
Stages of concern	Seven stages describing what a person worries about as they meet a change, from unconcerned to refocusing (Hall & Hord, 2014). R20 A model of how people's concerns about a change shift, from information and personal impact towards consequences and refinement (Hall & Hord, 2014).	R19Adoption and change in schoolsThe reluctant adopter · slide 28 (opens in a new tab) R20Communicating change to schoolsLead with the teacher · slide 20 (opens in a new tab)	Hall, G. E., & Hord, S. M. (2014). Implementing change: Patterns, principles, and potholes (4th ed.). Pearson. pearson.com American Institutes for Research. (n.d.). Stages of concern: Concerns-Based Adoption Model. Retrieved September 17, 2026, from air.org
Staging environment	A copy of the live system, made as similar to it as practical, where a release is rehearsed and checked before it goes to production.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com
Stakeholder interview	An interview with someone inside the organisation or its partners, held to learn the project's goals, constraints and politics before research with users begins.	R04Discovery researchFrom the field · not in the guide	Hall, E. (2019). Just enough research (2nd ed.). A Book Apart. abookapart.com
Stakeholder map	A diagram of the people and organisations involved in or affected by a service, arranged to show their relationships to one another and to the user.	R02Service design and systems thinkingFrom the field · not in the guide	Stickdorn, M., Hormess, M. E., Lawrence, A., & Schneider, J. (2018). This is service design doing: Applying service design thinking in the real world. O'Reilly Media. Stickdorn, M., & Schneider, J. (Eds.). (2010). This is service design thinking: Basics, tools, cases. BIS Publishers.
Standard deviation	Roughly the typical distance of values from their mean.	R18Basic statistics for educational technologistsDescribing data · slide 18 (opens in a new tab)	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Kunin, D., Guo, J., Devlin, T. D., & Xiang, D. (n.d.). Seeing theory: A visual introduction to probability and statistics. Brown University. seeing-theory.brown.edu
Standard error	The typical wobble of an estimate across repeated samples. Shrinks with the square root of sample size.	R18Basic statistics for educational technologistsUncertainty · slide 24 (opens in a new tab)	Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. doi Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican.
Standard error of measurement	An estimate of how far an observed score might sit from a student's true score, which makes a single score a range, not a point.	R18Basic statistics for educational technologistsWhere numbers come from · slide 52 (opens in a new tab)	Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
Standardised questionnaire	A questionnaire with fixed wording, response scale and scoring, whose reliability and validity have been tested, so that scores can be compared across studies and products.	R05Evaluative researchFrom the field · not in the guide	Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann. shop.elsevier.com
Statistical persona	A persona that adds to interviews a survey of at least 100, and ideally 500 or more, respondents (Laubheimer, 2020).	R06Design methods IThe misconception · slide 9 (opens in a new tab)	Laubheimer, P. (2020). 3 persona types: Lightweight, qualitative, and statistical. Nielsen Norman Group. nngroup.com Salminen, J., Guan, K., Jung, S.-G., & Jansen, B. J. (2021). A survey of 15 years of data-driven persona development. International Journal of Human–Computer Interaction, 37(18), 1685–1708. doi
Statistical significance	A p-value below a chosen threshold, conventionally 0.05. A convention, not a verdict.	R18Basic statistics for educational technologistsUncertainty · slide 25 (opens in a new tab)	Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. doi Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. doi Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi
Status page	A public page showing whether each part of a service is working, with updates on current incidents and planned maintenance.	R22Support and feedback loopsFrom the field · not in the guide
Statutory guidance	Guidance issued under a legal power, which schools must have regard to and need a good reason to depart from.	R20Communicating change to schoolsFrom the field · not in the guide
Steward	The person who defines a dataset's fields and maintains its quality day to day.	R14Data strategy and governanceNamed owners · slide 25 (opens in a new tab)	Privacy Technical Assistance Center. (2015). Data governance and stewardship (Rev. ed.). U.S. Department of Education. studentprivacy.ed.gov DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
Stock and flow	A stock is a quantity that builds up or drains over time, such as unmarked scripts; flows are the rates that fill and empty it. Stocks change only through their flows.	R02Service design and systems thinkingFrom the field · not in the guide	Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green. Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. McGraw-Hill.
Storage limitation	Keeping identifiable personal data no longer than its purpose requires (European Union, 2016).	R14Data strategy and governanceThe deletion date · slide 44 (opens in a new tab)	European Union. (2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation). Official Journal of the European Union, L 119, 1–88. eur-lex.europa.eu Personal Data Protection Commission. (n.d.). Data protection obligations. Retrieved September 17, 2026, from pdpc.gov.sg
Story mapping	Arranging user stories in two dimensions, with the steps of a user's activity across the top and detail beneath, so releases can be cut as slices across the whole journey.	R09Agile and Scrum in practiceFrom the field · not in the guide	Patton, J. (2014). User story mapping: Discover the whole story, build the right product. O'Reilly Media.
Story points	A unit for estimating the relative size of a piece of work, combining effort, complexity and uncertainty, instead of estimating in hours or days.	R09Agile and Scrum in practiceFrom the field · not in the guide	Cohn, M. (2005). Agile estimating and planning. Prentice Hall.
Storyboard	A sequence of drawn frames showing a person using a product in context over time, used to explore and explain a situation of use before any screens are designed.	R07Design methods IIFrom the field · not in the guide	Buxton, B. (2007). Sketching user experiences: Getting the design right and the right design. Morgan Kaufmann. books.google.com Truong, K. N., Hayes, G. R., & Abowd, G. D. (2006). Storyboarding: An empirical determination of best practices and effective guidelines. In Proceedings of the 6th Conference on Designing Interactive Systems (pp. 12–21). ACM. doi
Stratified randomisation	Random assignment carried out separately within groups defined by a key characteristic, such as school or prior attainment, so that the arms are balanced on it.	R13Experimentation and product analyticsFrom the field · not in the guide	Torgerson, D. J., & Torgerson, C. J. (2008). Designing randomised trials in health, education and the social sciences: An introduction. Palgrave Macmillan. doi
Street-level bureaucrat	A frontline public worker whose discretion shapes what policy becomes in practice (Lipsky, 2010).	R02Service design and systems thinkingBeyond your screen · slide 25 (opens in a new tab)	Lipsky, M. (2010). Street-level bureaucracy: Dilemmas of the individual in public services (30th anniversary expanded ed.). Russell Sage Foundation. (Original work published 1980) russellsage.org
Student engagement	A student's involvement in learning, usually split into behavioural, emotional and cognitive parts. Activity data captures mainly the behavioural part.	R15Learning analyticsFrom the field · not in the guide	Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. doi
Student retention	The share of students who continue on a course or at an institution from one period to the next. It is the outcome most early-warning systems aim to improve.	R15Learning analyticsFrom the field · not in the guide	Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase student success. In Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 267–270). ACM. doi Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). University of Chicago Press. doi
Style guide	An organisation's written rules for spelling, terms, punctuation and formatting, kept so that everything it publishes reads consistently.	R20Communicating change to schoolsFrom the field · not in the guide
Subgroup analysis	Estimating the effect separately for parts of the sample, such as disadvantaged pupils. Subgroups chosen after seeing the data produce many false positives.	R13Experimentation and product analyticsFrom the field · not in the guide	Education Endowment Foundation. (2022). Statistical analysis guidance for EEF evaluations. d2tic4wvo1iusb.cloudfront.net Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
Sunk cost fallacy	The tendency to keep investing in something because of what has already been spent on it, although that money or effort cannot be recovered whatever is decided next.	R01Digital product managementFrom the field · not in the guide	Arkes, H. R., & Blumer, C. (1985). The psychology of sunk cost. Organizational Behavior and Human Decision Processes, 35(1), 124–140. doi
Survey fatigue	Falling willingness to respond as people receive more surveys (Porter et al., 2004).	R05Evaluative researchQuestions that cannot lead · slide 33 (opens in a new tab) R22Support and feedback loopsClose the loop · slide 35 (opens in a new tab)	Porter, S. R., Whitcomb, M. E., & Weitzer, W. H. (2004). Multiple surveys of students and survey fatigue. New Directions for Institutional Research, 2004(121), 63–73. doi
SUS	The System Usability Scale: ten items giving one score from 0 to 100 (Brooke, 1996).	R05Evaluative researchScores with observations · slide 46 (opens in a new tab)	Brooke, J. (1996). SUS: A 'quick and dirty' usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189–194). Taylor & Francis. doi Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human–Computer Interaction, 24(6), 574–594. doi Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114–123. dl.acm.org
Sustainable pace	Working at a rate the team could keep up indefinitely, without regular overtime, on the ground that tired teams make more mistakes.	R09Agile and Scrum in practiceFrom the field · not in the guide	Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R. C., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D. (2001). Manifesto for Agile Software Development. agilemanifesto.org Beck, K., & Andres, C. (2004). Extreme programming explained: Embrace change (2nd ed.). Addison-Wesley.
Sustained duration	Professional learning activities that are ongoing throughout the school year, with 20 hours or more of contact time (Desimone & Garet, 2015).	R21Professional learning that changes practiceSpread it out · slide 27 (opens in a new tab)	Desimone, L. M., & Garet, M. S. (2015). Best practices in teachers' professional development in the United States. Psychology, Society, & Education, 7(3), 252–263. repositorio.ual.es Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yoon, K. S. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38(4), 915–945. doi Darling-Hammond, L., Hyler, M. E., & Gardner, M. (2017). Effective teacher professional development. Learning Policy Institute. doi
Swimlane diagram	A process diagram divided into parallel lanes, one for each person, team or system, so that every step shows who does it and every handover shows as a crossing.	R02Service design and systems thinkingFrom the field · not in the guide	Rummler, G. A., & Brache, A. P. (1990). Improving performance: How to manage the white space on the organization chart. Jossey-Bass.
Synthesis	The work of organising, pruning and filtering research data into a cohesive structure: deciding what the material shows, what it means and what the team should do about it (Kolko, 2010).	R06Design methods IThe spine · slide 15 (opens in a new tab)	Kolko, J. (2010). Abductive thinking and sensemaking: The drivers of design synthesis. Design Issues, 26(1), 15–28. doi
System archetype	A recurring structure of feedback loops that produces the same problem behaviour in many settings, such as fixes that fail or limits to growth.	R02Service design and systems thinkingFrom the field · not in the guide	Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. Doubleday/Currency. Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green.
System of record	The system treated as the authoritative source for a given piece of data. Where copies disagree, its value stands.	R14Data strategy and governanceFrom the field · not in the guide	DAMA International. (2017). DAMA-DMBOK: Data management body of knowledge (2nd ed.). Technics Publications. damadmbok.org
System prompt	Standing instructions set by a product's developer, usually hidden from the user, that fix the model's role, tone and limits in every conversation.	R16AI in learning productsFrom the field · not in the guide
Systems thinking	A way of explaining why changing one part of a service so often produces effects somewhere else, and why a well-meant change can make the problem it targets worse.	R02Service design and systems thinkingThe spine · slide 15 (opens in a new tab)	Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green. Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. Doubleday/Currency. OECD. (2017). Systems approaches to public sector challenges: Working with change. OECD Publishing. doi
T
T-shaped person	Someone with deep skill in one discipline, the upright of the T, and enough breadth and curiosity to work well with people from others, the crossbar.	R03Design thinking, used honestlyFrom the field · not in the guide	Brown, T. (2009). Change by design: How design thinking transforms organizations and inspires innovation. HarperBusiness.
t-test	A test of whether the difference between two means is larger than chance variation alone would plausibly produce.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25. doi Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE.
Tacit knowledge	Knowledge of their own work practice that people are not consciously aware of, and become aware of only in the doing (Holtzblatt & Beyer, 2013).	R04Discovery researchThe spine · slide 15 (opens in a new tab)	Holtzblatt, K., & Beyer, H. R. (2013). Contextual design. In M. Soegaard & R. F. Dam (Eds.), The encyclopedia of human-computer interaction (2nd ed.). Interaction Design Foundation. ixdf.org Polanyi, M. (1966). The tacit dimension. Doubleday.
Tag	A label on a ticket that records its underlying cause, so tickets can be counted by cause.	R22Support and feedback loopsTickets as research · slide 20 (opens in a new tab)	Government Digital Service. (2016). Set up and manage user support. GOV.UK Service Manual. gov.uk Consortium for Service Innovation. (2016). KCS v6 practices guide. library.serviceinnovation.org
Tame problem	A problem that can be stated clearly, has a definite end point and can be checked as solved or not, however hard it may be technically. The opposite of a wicked problem.	R03Design thinking, used honestlyFrom the field · not in the guide	Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. doi
Tapper study	A 1990 Stanford study in which people tapping out well-known songs predicted listeners would name about half, and listeners named three of 120, or 2.5% (Newton, 1990).	R20Communicating change to schoolsThe spine · slide 15 (opens in a new tab)	Newton, E. (1990). Overconfidence in the communication of intent: Heard and unheard melodies [Unpublished doctoral dissertation]. Stanford University. Heath, C., & Heath, D. (2007). Made to stick: Why some ideas survive and others die. Random House. heathbrothers.com Camerer, C., Loewenstein, G., & Weber, M. (1989). The curse of knowledge in economic settings: An experimental analysis. Journal of Political Economy, 97(5), 1232–1254. doi
Task scenario	A short, realistic situation given to a usability test participant that states a goal to reach, without naming the steps or the words used on screen.	R05Evaluative researchFrom the field · not in the guide	Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Wiley.
Task success	Whether a participant completes a test task, the usual measure of effectiveness.	R05Evaluative researchTest the task · slide 19 (opens in a new tab)	International Organization for Standardization. (2018). Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts (ISO Standard No. 9241-11:2018). iso.org Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann. shop.elsevier.com Sauro, J. (2011). What is a good task-completion rate? MeasuringU. measuringu.com
Taxonomy	The agreed, controlled set of tags and categories used to label research in a repository, so that different people file and find the same things in the same way.	R06Design methods IFrom the field · not in the guide	Rosala, M. (2024a). Research repositories for tracking UX research and growing your ResearchOps. Nielsen Norman Group. nngroup.com Rosala, M. (2024b). Why research repositories fail and how to get them right. Nielsen Norman Group. nngroup.com
Teacher self-efficacy	A teacher's belief in their own ability to bring about student engagement and learning, which affects how readily they try and persist with new practices.	R21Professional learning that changes practiceFrom the field · not in the guide	Tschannen-Moran, M., & Woolfolk Hoy, A. (2001). Teacher efficacy: Capturing an elusive construct. Teaching and Teacher Education, 17(7), 783–805. doi Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191–215. doi
Technical debt	The future cost of a shortcut taken now, paid as extra effort on every later change until it is repaid (Cunningham, 1992).	R11Software development for non-engineersThe spine · slide 13 (opens in a new tab)	Cunningham, W. (1992). The WyCash portfolio management system. In Addendum to the proceedings on object-oriented programming systems, languages, and applications (OOPSLA '92) (pp. 29–30). ACM. doi Kruchten, P., Nord, R. L., & Ozkaya, I. (2012). Technical debt: From metaphor to theory and practice. IEEE Software, 29(6), 18–21. doi Fowler, M. (2009, October 14). Technical debt quadrant. martinfowler.com. martinfowler.com
Technical debt quadrant	A sorting of technical debt two ways, reckless or prudent and deliberate or inadvertent; the useful distinction is between prudent and reckless debt (Fowler, 2009).	R11Software development for non-engineersTechnical debt · slide 25 (opens in a new tab)	Fowler, M. (2009, October 14). Technical debt quadrant. martinfowler.com. martinfowler.com Kruchten, P., Nord, R. L., & Ozkaya, I. (2012). Technical debt: From metaphor to theory and practice. IEEE Software, 29(6), 18–21. doi
Technology Acceptance Model	Davis's model in which perceived usefulness and perceived ease of use predict use of a system, and ease of use shapes whether a tool seems useful (Davis, 1989).	R19Adoption and change in schoolsEffort before training · slide 22 (opens in a new tab)	Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. doi Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. doi
Test automation	Having software run the tests, on every change and without a person starting them, so that a fault is reported within minutes of being introduced.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com
Test plan	The document agreed before a usability study that sets out its purpose, research questions, participants, tasks, measures and schedule.	R05Evaluative researchFrom the field · not in the guide	Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Wiley.
Test pyramid	A guide to balancing automated tests: many small unit tests, fewer broad ones (Fowler, 2012).	R11Software development for non-engineersKnowing it is broken · slide 37 (opens in a new tab)	Cohn, M. (2009). Succeeding with agile: Software development using Scrum. Addison-Wesley. mountaingoatsoftware.com Fowler, M. (2012, May 1). Test pyramid. martinfowler.com. martinfowler.com
Test-driven development (TDD)	Writing a failing automated test before writing the code that makes it pass, then tidying the code, in short repeated cycles.	R09Agile and Scrum in practiceFrom the field · not in the guide	Beck, K. (2003). Test-driven development: By example. Addison-Wesley. doi
Testbed	A real setting, such as a group of schools, where education technology is tried out and evaluated under everyday conditions, with teachers, developers and researchers working together.	R10Lean product developmentFrom the field · not in the guide	Batty, R., Florescu, A., Wong, A., & Sharples, M. (2019). EdTech testbeds: Models for improving evidence. Nesta. nesta.org.uk
The chasm	The gap between a product's enthusiastic early adopters and the pragmatic majority, who want proof, support and a complete solution before they commit.	R19Adoption and change in schoolsFrom the field · not in the guide	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press. Moore, G. A. (1991). Crossing the chasm: Marketing and selling technology products to mainstream customers. HarperBusiness.
The Three Ways	The principles said to underlie DevOps: fast flow of work from development to operations, fast feedback in the other direction, and a culture of continual experimentation and learning.	R12DevOps and continuous deliveryFrom the field · not in the guide	Kim, G., Humble, J., Debois, P., & Willis, J. (2016). The DevOps handbook: How to create world-class agility, reliability, and security in technology organizations. IT Revolution Press.
Thematic analysis	Analysis in six phases, moving from familiarisation with the data and initial coding to developing, reviewing, naming and reporting themes (Braun & Clarke, 2006).	R06Design methods ISynthesise together · slide 20 (opens in a new tab)	Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. doi Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. doi Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. doi
Theme	A pattern of shared meaning across data, organised around a central idea (Braun & Clarke, 2019).	R06Design methods ISynthesise together · slide 20 (opens in a new tab)	Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. doi Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. doi Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. doi
Theory of constraints	The view that a system's output is limited by its single tightest bottleneck, so improvement anywhere else changes little until that constraint is relieved.	R10Lean product developmentFrom the field · not in the guide	Goldratt, E. M., & Cox, J. (1984). The goal: A process of ongoing improvement. North River Press.
Thin slice	A small piece of a feature built through every layer, so users can use it on its own.	R09Agile and Scrum in practiceShip small · slide 20 (opens in a new tab)	Cohn, M. (2004). User stories applied: For agile software development. Addison-Wesley. mountaingoatsoftware.com Wake, B. (2003, August 17). INVEST in good stories, and SMART tasks. XP123. xp123.com
Thinking aloud	Asking participants to say what they are thinking while they work on a task (Nielsen, 2012).	R05Evaluative researchTest the task · slide 20 (opens in a new tab)	Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). MIT Press. doi Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137(2), 316–344. doi Nielsen, J. (2012). Thinking aloud: The #1 usability tool. Nielsen Norman Group. nngroup.com
Threat model	A structured account of what a system holds, who might attack it, how, and what would stop them, written before the build so that defences follow the risks.	R17Privacy, security and ethicsFrom the field · not in the guide	Shostack, A. (2014). Threat modeling: Designing for security. Wiley.
Throughput	The number of work items a team or process finishes in a given period, such as tests completed per term.	R10Lean product developmentFrom the field · not in the guide	Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Blue Hole Press. books.google.com Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com
Throwaway prototype	A prototype built only to answer a question and then discarded, with none of its code or materials carried into the product.	R07Design methods IIFrom the field · not in the guide	Tripp, S. D., & Bichelmeyer, B. (1990). Rapid prototyping: An alternative instructional design strategy. Educational Technology Research and Development, 38(1), 31–44. doi Floyd, C. (1984). A systematic look at prototyping. In R. Budde, K. Kuhlenkamp, L. Mathiassen, & H. Züllighoven (Eds.), Approaches to prototyping (pp. 1–18). Springer. doi
Ticket	The record of one request or report in a support system, holding who raised it, what was said and done, and its current status.	R22Support and feedback loopsFrom the field · not in the guide
Ticket deflection	The share of would-be support contacts that are resolved by self-service before a ticket is raised.	R22Support and feedback loopsFrom the field · not in the guide
Tiered support	An arrangement in which first-line staff handle common queries, and pass harder ones to second-line specialists and then to third-line engineers or the vendor.	R22Support and feedback loopsFrom the field · not in the guide
Time on task	How long a participant takes to complete a task, a measure of efficiency. R15 Time a student is estimated to have spent on an activity, usually inferred from gaps between logged actions (Kovanović et al., 2015).	R05Evaluative researchTest the task · slide 19 (opens in a new tab) R15Learning analyticsThe misconception · slide 10 (opens in a new tab)	International Organization for Standardization. (2018). Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts (ISO Standard No. 9241-11:2018). iso.org Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R. S., & Hatala, M. (2015). Penetrating the black box of time-on-task estimation. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (pp. 184–193). ACM. doi Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research (2nd ed.). Morgan Kaufmann. shop.elsevier.com
Time to restore	The time from a fault being reported to the user being able to work again.	R22Support and feedback loopsTime to fix · slide 38 (opens in a new tab)	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com DORA. (2026). DORA's software delivery performance metrics. Google Cloud. dora.dev
Timebox	A fixed maximum length of time given to an event or piece of work, which ends when the time runs out whether or not the work is finished.	R09Agile and Scrum in practiceFrom the field · not in the guide	Schwaber, K., & Sutherland, J. (2020). The Scrum Guide: The definitive guide to Scrum: The rules of the game. scrumguides.org
Toil	Operational work that is manual, repetitive, automatable and without lasting value, and that grows as the service grows. Reliability teams cap the share of their time spent on it.	R12DevOps and continuous deliveryFrom the field · not in the guide	Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site reliability engineering: How Google runs production systems. O'Reilly Media. sre.google
Token	The unit of text a language model reads and writes, typically a word or a fragment of one. Context limits and usage prices are counted in tokens.	R16AI in learning productsFrom the field · not in the guide	Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1715–1725. doi
Tone of voice	The consistent character of an organisation's writing, such as how formal, warm or direct it is, adjusted to the reader's situation.	R20Communicating change to schoolsFrom the field · not in the guide
Topic summary	A grouping that gathers everything said about one subject, as distinct from a theme, which is organised around a central idea (Braun & Clarke, 2019).	R06Design methods IThe misconception · slide 10 (opens in a new tab)	Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. doi Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. doi
Total cost of ownership (TCO)	The full cost of a product over its life, adding training, support, integration, maintenance and eventual replacement to the purchase or licence price.	R01Digital product managementFrom the field · not in the guide	Ellram, L. M. (1995). Total cost of ownership: An analytical approach for purchasing. International Journal of Physical Distribution & Logistics Management, 25(8), 4–23. doi
Touch target	The area of a screen that responds to a tap. Small or crowded targets cause errors, so accessibility guidance sets minimum sizes.	R08UX foundations and design systemsFrom the field · not in the guide	World Wide Web Consortium. (2023). Web Content Accessibility Guidelines (WCAG) 2.2 (W3C Recommendation). w3.org Budiu, R. (2022). Fitts's law and its applications in UX. Nielsen Norman Group. nngroup.com Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47(6), 381–391. doi
Touchpoint	Any moment where a user meets the service: a screen, a form, a message or a person.	R02Service design and systems thinkingThe spine · slide 12 (opens in a new tab)	Stickdorn, M., Hormess, M. E., Lawrence, A., & Schneider, J. (2018). This is service design doing: Applying service design thinking in the real world. O'Reilly Media. Polaine, A., Løvlie, L., & Reason, B. (2013). Service design: From insight to implementation. Rosenfeld Media.
Toyota Production System	The factory system Toyota developed from the 1950s, which names seven kinds of waste, among them overproduction, waiting and inventory (Ohno, 1988).	R10Lean product developmentThe spine · slide 15 (opens in a new tab)	Ohno, T. (1988). Toyota production system: Beyond large-scale production. Productivity Press. Womack, J. P., Jones, D. T., & Roos, D. (1990). The machine that changed the world. Rawson Associates. archive.org Krafcik, J. F. (1988). Triumph of the lean production system. Sloan Management Review, 30(1), 41–52.
TPACK	Technological pedagogical content knowledge: knowing how technology, teaching and subject content work together (Mishra & Koehler, 2006).	R21Professional learning that changes practiceThe teacher's own lesson · slide 21 (opens in a new tab)	Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. doi Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. doi
Tragedy of the commons	The pattern in which people each acting sensibly for themselves overuse a shared resource, such as teachers' time, until it is degraded for everyone.	R02Service design and systems thinkingFrom the field · not in the guide	Meadows, D. H. (2008). Thinking in systems: A primer (D. Wright, Ed.). Chelsea Green. Senge, P. M. (1990). The fifth discipline: The art and practice of the learning organization. Doubleday/Currency. Hardin, G. (1968). The tragedy of the commons. Science, 162(3859), 1243–1248. doi
Train-the-trainer	A model in which a small group is trained first and then trains colleagues in turn. It reaches many people cheaply, and content is diluted at each step.	R21Professional learning that changes practiceFrom the field · not in the guide	Hayes, D. (2000). Cascade training and teachers' professional development. ELT Journal, 54(2), 135–145. doi
Training data	The examples a model learns from. Their coverage, quality and biases shape what the model can do and whom it serves badly.	R16AI in learning productsFrom the field · not in the guide	National Institute of Standards and Technology. (2024). Artificial intelligence risk management framework: Generative artificial intelligence profile (NIST AI 600-1). U.S. Department of Commerce. doi Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. doi Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., III, & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92. doi
Training needs analysis	A study, made before training is designed, of what people must be able to do, what they can do now, and whether training is the right answer to the gap.	R21Professional learning that changes practiceFrom the field · not in the guide
Transfer	The use of something learned in training in the setting where it is meant to be used.	R21Professional learning that changes practiceThe misconception · slide 9 (opens in a new tab)	Showers, B., & Joyce, B. (1996). The evolution of peer coaching. Educational Leadership, 53(6), 12–16. ascd.org Joyce, B., & Showers, B. (2002). Student achievement through staff development (3rd ed.). Association for Supervision and Curriculum Development. books.google.com
Transformer	The neural network design behind current language models. It uses attention to weigh every part of the input against every other part when producing each output.	R16AI in learning productsFrom the field · not in the guide	Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.
Tree testing	A test of a menu structure in which participants are shown only the text hierarchy and asked where they would look for given items.	R05Evaluative researchFrom the field · not in the guide	Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com
Triage	Sorting a new report by severity and type, and deciding who handles it and how fast.	R22Support and feedback loopsThe spine · slide 14 (opens in a new tab)	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com
Trialability	How easily an innovation can be tried on a small scale before committing to it.	R19Adoption and change in schoolsFrom the field · not in the guide	Rogers, E. M. (2003). Diffusion of innovations (5th ed.). Free Press.
Triangulation	Checking a finding against more than one method, data source or researcher, so that a weakness in any one of them is less likely to mislead.	R04Discovery researchFrom the field · not in the guide	Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). McGraw-Hill. Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and practice (4th ed.). SAGE.
Trunk-based development	A way of working in which developers merge small changes into one shared main branch at least daily, keeping any other branches short-lived.	R12DevOps and continuous deliveryFrom the field · not in the guide	Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The science of lean software and DevOps: Building and scaling high performing technology organizations. IT Revolution. itrevolution.com Fowler, M. (2024). Continuous integration. martinfowler.com. (Original work published 2000) martinfowler.com
Tuning protocol	A structured, timed conversation among educators in which a presenter shares work and a question, then listens silently while colleagues give feedback.	R07Design methods IIFrom the field · not in the guide	McDonald, J. P., Mohr, N., Dichter, A., & McDonald, E. C. (2013). The power of protocols: An educator's guide to better practice (3rd ed.). Teachers College Press.
Two sigma problem	Bloom's (1984) challenge to find group teaching as effective as the two-standard-deviation gain he reported for tutoring.	R16AI in learning productsDate the evidence · slide 40 (opens in a new tab)	Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16. doi von Hippel, P. T. (2024). Two-sigma tutoring: Separating science fiction from science fact. Education Next, 24(2). educationnext.org VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. doi
Two-by-two matrix	A grid made by crossing two dimensions, giving four quadrants into which cases, users or ideas are placed to show how they differ.	R06Design methods IFrom the field · not in the guide	Lowy, A., & Hood, P. (2004). The power of the 2 × 2 matrix: Using 2 × 2 thinking to solve business problems and make better decisions. Jossey-Bass.
Two-step flow of communication	The finding that messages often reach people through opinion leaders, who take them from the media and pass them on with their own interpretation.	R20Communicating change to schoolsFrom the field · not in the guide	Katz, E., & Lazarsfeld, P. F. (1955). Personal influence: The part played by people in the flow of mass communications. Free Press.
Two-way communication	Communication in which the audience can reply and the sender listens and adjusts, as opposed to one-way broadcast.	R20Communicating change to schoolsFrom the field · not in the guide	Grunig, J. E., & Hunt, T. (1984). Managing public relations. Holt, Rinehart and Winston.
Twyman's law	Any figure that looks interesting or different is usually wrong, so the more a result looks like a breakthrough, the more checking it needs (Kohavi et al., 2020).	R13Experimentation and product analyticsThe spine · slide 15 (opens in a new tab)	Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. doi
Type I and Type II errors	A Type I error is concluding there is an effect when there is none. A Type II error is failing to detect an effect that is real.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. doi Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231, 289–337. doi
U
UMUX-LITE	A two-item usability questionnaire that tracks SUS closely (Lewis et al., 2013).	R05Evaluative researchScores with observations · slide 46 (opens in a new tab)	Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013). UMUX-LITE: When there's no time for the SUS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2099–2102). ACM. doi Finstad, K. (2010). The usability metric for user experience. Interacting with Computers, 22(5), 323–327. doi
Unanticipated consequences	Effects of a purposive action that nobody planned, as when people inside a system adapt to a change and their adaptations cancel it out (Merton, 1936).	R02Service design and systems thinkingThe spine · slide 15 (opens in a new tab)	Merton, R. K. (1936). The unanticipated consequences of purposive social action. American Sociological Review, 1(6), 894–904. doi Sterman, J. D. (2006). Learning from evidence in a complex world. American Journal of Public Health, 96(3), 505–514. doi
Unfreeze–change–refreeze	Lewin's three-stage account of change: loosen the current way of working, move to the new one, then stabilise it so people do not drift back.	R19Adoption and change in schoolsFrom the field · not in the guide	Lewin, K. (1947). Frontiers in group dynamics: Concept, method and reality in social science; social equilibria and social change. Human Relations, 1(1), 5–41. doi
Unified Theory of Acceptance and Use of Technology	A model combining eight earlier models of technology acceptance, with four core determinants: performance expectancy, effort expectancy, social influence and facilitating conditions (Venkatesh et al., 2003).	R19Adoption and change in schoolsEffort before training · slide 21 (opens in a new tab)	Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. doi Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. doi
Unique pupil number (UPN)	England's 13-character pupil identifier, expected to stay with a pupil throughout their school career, never reissued, and not to be used for any purpose unrelated to education (Department for Education, 2019).	R14Data strategy and governanceIdentifiers first · slide 38 (opens in a new tab)	Department for Education. (2019). Unique pupil number (UPN) (Guide version 1.2). assets.publishing.service.gov.uk
Unit test	An automated check of one small piece of code, usually run in seconds on every change.	R11Software development for non-engineersKnowing it is broken · slide 37 (opens in a new tab)	Fowler, M. (2012, May 1). Test pyramid. martinfowler.com. martinfowler.com Cohn, M. (2009). Succeeding with agile: Software development using Scrum. Addison-Wesley. mountaingoatsoftware.com
Usability	The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use (International Organization for Standardization, 2018).	R05Evaluative researchThe spine · slide 15 (opens in a new tab)	International Organization for Standardization. (2018). Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts (ISO Standard No. 9241-11:2018). iso.org
Usability test	A test that samples people, tasks and conditions: it shows how some people did some tasks in one setting, and holds for what was specified and no further.	R05Evaluative researchThe misconception · slide 8 (opens in a new tab)	International Organization for Standardization. (2018). Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts (ISO Standard No. 9241-11:2018). iso.org Rohrer, C. (2022). When to use which user-experience research methods. Nielsen Norman Group. nngroup.com
User acceptance testing (UAT)	Testing by the customer or the intended users, before go-live, of whether a system does what they need in their own setting. Often a formal condition of accepting and paying for delivered work.	R11Software development for non-engineersFrom the field · not in the guide	Myers, G. J. (1979). The art of software testing. Wiley.
User experience (UX)	A person's perceptions and responses that result from using, or expecting to use, a product, system or service.	R08UX foundations and design systemsFrom the field · not in the guide	Norman, D. (2013). The design of everyday things (Rev. and expanded ed.). Basic Books. basicbooks.com International Organization for Standardization. (2019). Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems (ISO 9241-210:2019).
User story	A short description of something a user needs, written to prompt conversation and confirmed by acceptance criteria (Jeffries, 2001).	R09Agile and Scrum in practiceShip small · slide 20 (opens in a new tab)	Jeffries, R. (2001). Essential XP: Card, conversation, confirmation. ronjeffries.com. ronjeffries.com Cohn, M. (2004). User stories applied: For agile software development. Addison-Wesley. mountaingoatsoftware.com Wake, B. (2003, August 17). INVEST in good stories, and SMART tasks. XP123. xp123.com
UX writing	Writing the words inside an interface, such as button labels, error messages and instructions, so that users can complete their task. The short pieces of text are often called microcopy.	R08UX foundations and design systemsFrom the field · not in the guide	Podmajersky, T. (2019). Strategic writing for UX: Drive engagement, conversion, and retention with every word. O'Reilly Media.
V
Validated learning	Progress shown by evidence about users, rather than by what has been built.	R10Lean product developmentCount learning · slide 38 (opens in a new tab)	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com Ries, E. (2009, August 3). Minimum viable product: A guide. Startup Lessons Learned. startuplessonslearned.com
Validity	The degree to which evidence supports the intended meaning and use of scores.	R18Basic statistics for educational technologistsWhere numbers come from · slide 51 (opens in a new tab)	Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. doi Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
Value demand	Demand that a service exists to meet, as distinct from the failure demand it creates by getting something wrong (Seddon, 2003).	R02Service design and systems thinkingClose the loop · slide 43 (opens in a new tab)	Seddon, J. (2003). Freedom from command and control: A better way to make the work work. Vanguard Education.
Value hypothesis	The assumption that people will find a product valuable once they use it. It is tested by watching what early users do, not by asking what they think.	R10Lean product developmentFrom the field · not in the guide	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com
Value proposition	A statement of the benefit a product offers a particular group of users, and why they should choose it over what they do now.	R01Digital product managementFrom the field · not in the guide	Osterwalder, A., Pigneur, Y., Bernarda, G., & Smith, A. (2014). Value proposition design: How to create products and services customers want. Wiley.
Value stream mapping	Drawing every step, hand-off and wait between a request and its delivery, with the time each takes, to show where work sits idle and which steps add nothing.	R10Lean product developmentFrom the field · not in the guide	Womack, J. P., & Jones, D. T. (1996). Lean thinking: Banish waste and create wealth in your corporation. Simon & Schuster. lean.org Rother, M., & Shook, J. (1999). Learning to see: Value-stream mapping to create value and eliminate muda. Lean Enterprise Institute.
Vanity metric	A number that rises with almost any activity and does not inform a decision, such as total logins.	R10Lean product developmentCount learning · slide 38 (opens in a new tab)	Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation to create radically successful businesses. Crown Business. theleanstartup.com Schonfeld, E. (2011, September 24). Founder stories: Eric Ries on "vanity metrics" and "success theater". TechCrunch. techcrunch.com
Variance	The average squared distance of values from their mean. The standard deviation is its square root, and 'variance explained' describes how much of it a model accounts for.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Spiegelhalter, D. (2019). The art of statistics: Learning from data. Pelican. Wheelan, C. (2013). Naked statistics: Stripping the dread from the data. W. W. Norton. Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE.
Velocity	The amount of work, usually in story points, that a team finishes in a sprint. Teams average it over recent sprints to forecast how much they can take on next.	R09Agile and Scrum in practiceFrom the field · not in the guide	Cohn, M. (2005). Agile estimating and planning. Prentice Hall.
Vendor lock-in	A position in which moving away from a supplier's product would cost so much, in data migration, retraining or contract penalties, that the buyer stays whatever the price or quality.	R01Digital product managementFrom the field · not in the guide	Hopson, M., McFadden, V., Refoy, R., & Rouault, A. (Eds.). (2020). De-risking government technology: Federal agency field guide. 18F, U.S. General Services Administration. guides.18f.gov Shapiro, C., & Varian, H. R. (1999). Information rules: A strategic guide to the network economy. Harvard Business School Press. doi
Verbatim	A participant's exact words, quoted without paraphrase. It is kept as evidence so that readers can judge the researcher's interpretation for themselves.	R04Discovery researchFrom the field · not in the guide	Portigal, S. (2023). Interviewing users: How to uncover compelling insights (2nd ed.). Rosenfeld Media. rosenfeldmedia.com
Version control	A system that records every change to a set of files, who made it and why, so that earlier versions can be recovered and several people can work on the same code.	R11Software development for non-engineersFrom the field · not in the guide	Loeliger, J., & McCullough, M. (2012). Version control with Git: Powerful tools and techniques for collaborative software development (2nd ed.). O'Reilly Media.
Virtual machine	A software imitation of a whole computer, with its own operating system, running alongside others on shared physical hardware. Cloud servers are usually virtual machines.	R12DevOps and continuous deliveryFrom the field · not in the guide	Popek, G. J., & Goldberg, R. P. (1974). Formal requirements for virtualizable third generation architectures. Communications of the ACM, 17(7), 412–421. doi
Visual hierarchy	The ordering of elements on a screen by size, weight, colour and position so that the eye meets the most important thing first.	R08UX foundations and design systemsFrom the field · not in the guide	Lidwell, W., Holden, K., & Butler, J. (2010). Universal principles of design (Rev. and updated ed.). Rockport Publishers.
Voice of the customer (VoC)	A programme that gathers what users say across support tickets, surveys, interviews and reviews, and reports it to the people who decide what to build.	R22Support and feedback loopsFrom the field · not in the guide	Griffin, A., & Hauser, J. R. (1993). The voice of the customer. Marketing Science, 12(1), 1–27. doi
Volunteer bias	The difference between people who choose to take part in research and those who do not (Rosenthal & Rosnow, 1975).	R04Discovery researchBeyond the keen · slide 31 (opens in a new tab)	Rosenthal, R., & Rosnow, R. L. (1975). The volunteer subject. Wiley. books.google.com Price, P. C., Jhangiani, R. S., & Chiang, I. A. (2015). Research methods in psychology (2nd Canadian ed.). BCcampus. opentextbc.ca Government Digital Service. (2020). Find user research participants. GOV.UK Service Manual. gov.uk
Vulnerability	A weakness in software, configuration or process that an attacker could use. One that is attacked before the maker has a fix is a zero-day.	R17Privacy, security and ethicsFrom the field · not in the guide	Bilge, L., & Dumitraş, T. (2012). Before we knew it: An empirical study of zero-day attacks in the real world. In Proceedings of the 2012 ACM Conference on Computer and Communications Security (pp. 833–844). ACM. doi Joint Task Force Transformation Initiative. (2012). Guide for conducting risk assessments (NIST Special Publication 800-30, Rev. 1). National Institute of Standards and Technology. doi
Vulnerability disclosure	A published route by which outsiders can report a security flaw and have it handled without legal threat. A bug bounty adds payment for valid reports.	R17Privacy, security and ethicsFrom the field · not in the guide	International Organization for Standardization & International Electrotechnical Commission. (2018). Information technology — Security techniques — Vulnerability disclosure (ISO/IEC 29147:2018).
W
WAI-ARIA	A W3C specification of attributes added to web code to tell assistive technology the role, name and state of interface parts that plain HTML cannot describe.	R08UX foundations and design systemsFrom the field · not in the guide	World Wide Web Consortium. (2023). Accessible Rich Internet Applications (WAI-ARIA) 1.2 (W3C Recommendation).
Waitlist control	A control group that receives the intervention after the trial's outcomes have been measured, so that no participant is denied it.	R13Experimentation and product analyticsFrom the field · not in the guide	Torgerson, D. J., & Torgerson, C. J. (2008). Designing randomised trials in health, education and the social sciences: An introduction. Palgrave Macmillan. doi
Warm and cool feedback	A protocol convention: warm feedback names what works in a piece of work; cool feedback raises questions and gaps between the work and its goals.	R07Design methods IIFrom the field · not in the guide	McDonald, J. P., Mohr, N., Dichter, A., & McDonald, E. C. (2013). The power of protocols: An educator's guide to better practice (3rd ed.). Teachers College Press.
Waterfall	Delivering software in one pass through sequential phases of requirements, design, build, test and release, each completed and signed off before the next begins.	R09Agile and Scrum in practiceFrom the field · not in the guide	Royce, W. W. (1970). Managing the development of large software systems. In Proceedings of IEEE WESCON (pp. 1–9). IEEE. Larman, C., & Basili, V. R. (2003). Iterative and incremental development: A brief history. Computer, 36(6), 47–56. doi
Web Content Accessibility Guidelines (WCAG)	The World Wide Web Consortium's accessibility standard; version 2.2 sets minimums such as 4.5:1 contrast for normal text and 24 by 24 CSS pixel targets (World Wide Web Consortium, 2023). R05 The Web Content Accessibility Guidelines; version 2.2 became a W3C Recommendation in October 2023, and meeting it does not address every user need (World Wide Web Consortium, 2023).	R08UX foundations and design systemsThe front of the room · slide 38 (opens in a new tab) R05Evaluative researchThe real pace, the real room · slide 41 (opens in a new tab)	World Wide Web Consortium. (2023). Web Content Accessibility Guidelines (WCAG) 2.2 (W3C Recommendation). w3.org W3C Web Accessibility Initiative. (2024). Involving users in evaluating web accessibility. w3.org
Weighted shortest job first (WSJF)	A way of ordering work by dividing each item's cost of delay by its duration, so that short, valuable jobs are done first.	R01Digital product managementFrom the field · not in the guide	Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com Scaled Agile, Inc. (n.d.). WSJF. SAFe Framework. framework.scaledagile.com Arnold, J. (n.d.). Cost of delay divided by duration. Black Swan Farming. blackswanfarming.com
Wicked problem	A problem with no single owner, no agreed definition and no clear stopping point (Rittel & Webber, 1973). R03 A problem whose formulation is itself the problem, with no single owner or stopping point (Rittel & Webber, 1973).	R02Service design and systems thinkingPolicy as material · slide 38 (opens in a new tab) R03Design thinking, used honestlyThe spine · slide 16 (opens in a new tab)	Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. doi OECD. (2017). Systems approaches to public sector challenges: Working with change. OECD Publishing. doi Buchanan, R. (1992). Wicked problems in design thinking. Design Issues, 8(2), 5–21. doi
Wireframe	A plain line drawing of a screen that shows what goes where and in what order of importance, leaving out colour, imagery and final wording.	R07Design methods IIFrom the field · not in the guide	Garrett, J. J. (2011). The elements of user experience: User-centered design for the web and beyond (2nd ed.). New Riders. Warfel, T. Z. (2009). Prototyping: A practitioner's guide. Rosenfeld Media.
Wizard of Oz test	A test in which a hidden person plays the part of a system (Kelley, 1984). R10 A test in which a hidden person simulates a system that users believe is automated (Kelley, 1984).	R07Design methods IIMatch the fidelity · slide 26 (opens in a new tab) R10Lean product developmentThe smallest test · slide 26 (opens in a new tab)	Kelley, J. F. (1984). An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems, 2(1), 26–41. doi Dahlbäck, N., Jönsson, A., & Ahrenberg, L. (1993). Wizard of Oz studies: Why and how. Knowledge-Based Systems, 6(4), 258–266. doi Kelley, J. F. (2018). Wizard of Oz (WoZ): A yellow brick journey. Journal of Usability Studies, 13(3), 119–124. uxpajournal.org
Work in progress (WIP)	Work that has started and not yet finished, including pilots with no decision yet.	R10Lean product developmentWork in progress · slide 31 (opens in a new tab)	Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Blue Hole Press. books.google.com Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com Poppendieck, M., & Poppendieck, T. (2003). Lean software development: An agile toolkit. Addison-Wesley. informit.com
Work in progress (WIP) limit	The most items allowed in one stage of work at a time.	R09Agile and Scrum in practiceMake work visible · slide 26 (opens in a new tab)	Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Blue Hole Press. books.google.com Reinertsen, D. G. (2009). The principles of product development flow: Second generation lean product development. Celeritas Publishing. lpd2.com
Workaround	A way of reducing or avoiding the effect of a fault that lets the user carry on before the cause has been fixed.	R22Support and feedback loopsFrom the field · not in the guide	AXELOS. (2019). ITIL Foundation: ITIL 4 edition. TSO. axelos.com
Working memory	The small, temporary store in which people hold and work on information.	R08UX foundations and design systemsLess to remember · slide 20 (opens in a new tab)	Cowan, N. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral and Brain Sciences, 24(1), 87–114. doi Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81–97. doi
Z
z-score	A value expressed as the number of standard deviations it lies above or below the mean, which lets results on different scales be compared.	R18Basic statistics for educational technologistsFrom the field · not in the guide	Wheelan, C. (2013). Naked statistics: Stripping the dread from the data. W. W. Norton. Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). SAGE.
Zero trust	A security model that trusts no user or device because of its network location, and checks identity and device on each request for a resource.	R17Privacy, security and ethicsFrom the field · not in the guide	Rose, S., Borchert, O., Mitchell, S., & Connelly, S. (2020). Zero trust architecture (NIST Special Publication 800-207). National Institute of Standards and Technology. doi
Zero-downtime release	A release that moves users from the old version to the new one without the service becoming unavailable, so it need not wait for an overnight or weekend slot.	R12DevOps and continuous deliveryFrom the field · not in the guide	Humble, J., & Farley, D. (2010). Continuous delivery: Reliable software releases through build, test, and deployment automation. Addison-Wesley. oreilly.com

Nearby

The other glossary, and the series behind this one

Singapore