Gerald Ajam

Library / Glossary / The context glossary

Programme evaluation glossary

30 terms from C22, Programme evaluation — each defined in the primer's own words. Every term the primer teaches links to the slide that teaches it.

The whole context glossary

C22 · Evidence and change

Programme evaluation

“Evaluation is research done on a programme.”

Every term below is defined in the words of programme evaluation, primer C22 of understanding the context of education, and opens the primer at the slide where it is taught. 3 terms are also defined by another primer in the series; where the two differ, both wordings are given. The whole context glossary holds all of them together.

TermDefinitionReferred to inRead further
A
Attribution against contribution

Attribution claims a share of the observed change for the programme and needs a counterfactual. Contribution claims the programme was one cause among several, none sufficient alone, and is established by testing the theory and the rival explanations (Mayne, 2012).

  • Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270–280. doi
  • Treasury Board of Canada Secretariat. (2021, March 22). Theory-based approaches to evaluation: Concepts and practices. Government of Canada. canada.ca
C
CIPP model

Stufflebeam's model of four kinds of evaluation, each answering one question: context, "What needs to be done?"; input, "How should it be done?"; process, "Is it being done?"; product, "Did it succeed?" (Stufflebeam, 2007).

  • Stufflebeam, D. L. (2007). CIPP evaluation model checklist (2nd ed.). The Evaluation Center, Western Michigan University. files.wmich.edu
  • Stufflebeam, D. L. (2003). The CIPP model for evaluation. In T. Kellaghan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 31–62). Springer. doi
Context–mechanism–outcome configurations

The unit of a realist evaluation: a statement of how a programme's resources trigger a mechanism, among whom and in which conditions, to produce a given outcome. Summarised as "context + mechanism = outcome" (Pawson & Tilley, 2004).

  • Pawson, R., & Tilley, N. (1997). Realistic evaluation. Sage.
  • Pawson, R., & Tilley, N. (2004). Realist evaluation. cnxus.org
Counterfactual

What would have happened to the same people without the programme. It cannot be observed, so it is estimated from a comparison or control group. Primer C21 covers the designs used to do this.

C21 What would have happened to the same students without the programme. That world cannot be observed, so every causal design is a way of building a stand-in for it.

  • Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270–280. doi
  • Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
  • Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960. doi
D
Developmental evaluation

Evaluation of an innovation that is still changing, where the model is not settled and may never be. It supplies continuous feedback rather than a verdict (Patton, 1994; Gamble, 2008).

  • Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15(3), 311–319. doi
  • Gamble, J. A. A. (2008). A developmental evaluation primer. The J. W. McConnell Family Foundation. mcconnellfoundation.ca
E
Economic evaluation and cost-effectiveness

Analysis of what a programme consumes as well as what it achieves. Cost-effectiveness compares outcomes per unit of cost against alternatives; cost-benefit analysis puts a monetary value on the benefits. Costs are built up by the ingredients method, which lists every resource a programme needs (Hollands et al., 2020).

  • Hollands, F., Pratt-Williams, J., & Shand, R. (2020). Cost analysis standards & guidelines 1.0. Cost Analysis in Practice (CAP) Project. eed.communities.ed.gov
  • Levin, H. M., McEwan, P. J., Belfield, C., Bowden, A. B., & Shand, R. (2018). Economic evaluation in education: Cost-effectiveness and cost-benefit analysis (3rd ed.). Sage.
Empowerment evaluation

A participatory approach in which the evaluator supports programme staff and participants to evaluate their own work, with capacity and ownership as intended outcomes (Fetterman, 1994).

  • Fetterman, D. M. (1994). Empowerment evaluation. Evaluation Practice, 15(1), 1–15. doi
  • Fetterman, D. M., Kaftarian, S. J., & Wandersman, A. (Eds.). (1996). Empowerment evaluation: Knowledge and tools for self-assessment and accountability. Sage.
Evaluability assessment

A short, structured check on whether a programme can usefully be evaluated: are its goals clear, is its theory plausible, are the data available, and would anyone act on the answer (Wholey, 1987; Craig & Campbell, 2015)?

  • Wholey, J. S. (1987). Evaluability assessment: Developing program theory. New Directions for Program Evaluation, 1987(33), 77–92. doi
  • Craig, P., & Campbell, M. (2015). Evaluability assessment: A systematic approach to deciding whether and how to evaluate programmes and policies [Working paper]. What Works Scotland. whatworksscotland.ac.uk
Evaluation against research

Research sets its own questions and aims at knowledge that generalises; an evaluation answers questions set by a programme and its users, about that programme, and ends in a judgement and usually a recommendation (Levin-Rozalis, 2003).

  • Levin-Rozalis, M. (2003). Evaluation and research: Differences and similarities. The Canadian Journal of Program Evaluation, 18(2), 1–31. doi
Evaluation of professional development

Evaluation of training for teachers and staff. Guskey (2002) sets out five levels: reactions, learning, organisation support and change, use of new knowledge and skills, and student learning outcomes.

  • Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45–51. ascd.org
  • Guskey, T. R. (2014). Planning professional learning. Educational Leadership, 71(8), 10–16. tguskey.com
Evaluative rubrics and reasoning

The explicit path from evidence to a value conclusion: criteria saying what matters, and standards saying what weak, adequate and strong performance would look like on each, agreed before the data arrive (Davidson, 2005).

  • Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Sage.
F
Formative and summative evaluation

Formative evaluation is done to improve something while it is still being shaped; summative evaluation is done to reach a verdict about it. Scriven (1996) argues that the same activity can be either, depending on the context it is used in.

  • Scriven, M. (1996). Types of evaluation and types of evaluator. Evaluation Practice, 17(2), 151–161. doi
  • Scriven, M. (1991b). Evaluation thesaurus (4th ed.). Sage.
  • Gamble, J. A. A. (2008). A developmental evaluation primer. The J. W. McConnell Family Foundation. mcconnellfoundation.ca
G
Goal-free evaluation

Evaluation in which the evaluator is deliberately kept from the programme's stated goals, so as to record what it is actually doing and to catch effects nobody intended, including harms (Scriven, 1991b, quoted in Youker, 2024).

  • Scriven, M. (1991a). Prose and cons about goal-free evaluation. Evaluation Practice, 12(1), 55–62. doi
  • Scriven, M. (1991b). Evaluation thesaurus (4th ed.). Sage.
  • Youker, B. W. (2024). Scriven’s goal-free evaluation. Journal of MultiDisciplinary Evaluation, 20(47), 28–31. jmde.com
K
Kirkpatrick's four levels

A level model used for courses, training and edtech that runs from reaction to learning to behaviour to results: a useful ladder and a dangerous shortcut (Kirkpatrick & Kirkpatrick, 2006).

  • Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler.
  • Alliger, G. M., Tannenbaum, S. I., Bennett, W., Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50(2), 341–358. doi
  • Bates, R. (2004). A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341–347. doi
L
Logic model

A diagram of a programme as a chain: resources, activities, outputs, immediate outcomes, intermediate outcomes and ultimate outcomes (Treasury Board of Canada Secretariat, 2021). It lists the boxes but does not explain the arrows.

  • Treasury Board of Canada Secretariat. (2021, March 22). Theory-based approaches to evaluation: Concepts and practices. Government of Canada. canada.ca
  • W. K. Kellogg Foundation. (2004). Logic model development guide. W. K. Kellogg Foundation.
M
Merit and worth

Merit is how good something is in itself; worth is how valuable it is to a particular set of people in meeting their needs. Stufflebeam (2007) adds probity, meaning integrity and honesty, and significance, meaning importance beyond its own setting.

  • Stufflebeam, D. L. (2007). CIPP evaluation model checklist (2nd ed.). The Evaluation Center, Western Michigan University. files.wmich.edu
  • Scriven, M. (1991b). Evaluation thesaurus (4th ed.). Sage.
  • Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Sage.
Meta-evaluation

Evaluation of an evaluation, against professional standards. The Program Evaluation Standards devote a whole category to it, and Stufflebeam (2001) argued that it is an obligation rather than an extra.

  • Stufflebeam, D. L. (2001). The metaevaluation imperative. American Journal of Evaluation, 22(2), 183–209. doi
  • Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage.
  • Joint Committee on Standards for Educational Evaluation. (2018). Checklist of the program evaluation standards statements. files.wmich.edu
N
Needs assessment

A study of what the people a programme serves actually need, which supplies the criteria against which worth is later judged. In the CIPP model this is context evaluation, asking "What needs to be done?" (Stufflebeam, 2007).

  • Stufflebeam, D. L. (2007). CIPP evaluation model checklist (2nd ed.). The Evaluation Center, Western Michigan University. files.wmich.edu
  • Stufflebeam, D. L. (2003). The CIPP model for evaluation. In T. Kellaghan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 31–62). Springer. doi
O
OECD evaluation criteria

Six criteria for development evaluation: relevance, coherence, effectiveness, efficiency, impact and sustainability, which together "provide a normative framework used to determine the merit or worth" of an intervention (OECD, n.d.).

  • OECD. (n.d.). Evaluation criteria. Retrieved September 18, 2026, from oecd.org
Outcome and impact evaluation

Outcome evaluation asks what changed among those the programme reached. Impact evaluation asks how much of that change would not have happened otherwise, which requires a credible comparison.

  • Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ, 374, n2061. doi
  • Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.). Sage.
P
Pilot evaluation

The evaluation of a small first run, normally to test whether the programme can be delivered and whether an outcome study is feasible. A pilot with volunteers, extra support and no comparison cannot tell you whether the programme works.

  • Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ, 374, n2061. doi
Process and implementation evaluation

Evaluation of whether a programme was delivered as intended and how it was experienced, covering fidelity, quality, reach and the contextual factors behind variation in outcomes (Moore et al., 2015).

  • Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., Moore, L., O’Cathain, A., Tinati, T., Wight, D., & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. BMJ, 350, h1258. doi
  • Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ, 374, n2061. doi
Program Evaluation Standards

Thirty standards in five categories: utility, feasibility, propriety, accuracy and accountability. They are the field's nearest thing to a shared code, and utility comes first (Joint Committee on Standards for Educational Evaluation, 2018).

  • Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage.
  • Joint Committee on Standards for Educational Evaluation. (2018). Checklist of the program evaluation standards statements. files.wmich.edu
Programme evaluation

The systematic assessment of a programme's merit, worth, probity and significance, for people who have to decide something about it (Stufflebeam, 2007). It is a field with its own models, standards and professional bodies.

C21 Judging the merit, worth or value of a programme for the people who must decide about it. It has its own discipline, standards and models, covered in Programme evaluation (C22).

  • Stufflebeam, D. L. (2007). CIPP evaluation model checklist (2nd ed.). The Evaluation Center, Western Michigan University. files.wmich.edu
  • Pawson, R., & Tilley, N. (1997). Realistic evaluation. Sage.
  • Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Sage.
Programme theory

The account of how and why a programme is supposed to produce its effects, including the mechanisms it relies on and the conditions it needs (Weiss, 1997b; Coryn et al., 2011).

  • Weiss, C. H. (1997b). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501–524. doi
  • Weiss, C. H. (1997a). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 1997(76), 41–55. doi
  • Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226. doi
R
Realist evaluation

Evaluation that asks "What works for whom in what circumstances and in what respects, and how?", on the claim that an outcome follows from a mechanism firing in a context (Pawson & Tilley, 2004).

C21 An approach that asks what works for whom in what circumstances, treating context as part of the finding rather than as noise (Pawson & Tilley, 1997).

  • Pawson, R., & Tilley, N. (1997). Realistic evaluation. Sage.
  • Pawson, R., & Tilley, N. (2004). Realist evaluation. cnxus.org
S
Stakeholder and participatory evaluation

Approaches that involve the people with a stake in a programme in framing questions, interpreting findings or running the evaluation. Involvement is the factor most consistently linked to whether evaluations get used (Johnson et al., 2009).

  • Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., & Volkov, B. (2009). Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30(3), 377–410. doi
  • Fetterman, D. M. (1994). Empowerment evaluation. Evaluation Practice, 15(1), 1–15. doi
T
Theory of change

The explanation of why each step in the chain should follow from the one before, with its assumptions, risks and context made explicit (Treasury Board of Canada Secretariat, 2021).

  • Treasury Board of Canada Secretariat. (2021, March 22). Theory-based approaches to evaluation: Concepts and practices. Government of Canada. canada.ca
  • Weiss, C. H. (1997b). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501–524. doi
  • Weiss, C. H. (1997a). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 1997(76), 41–55. doi
U
Use and influence of evaluation

What an evaluation actually changes: a decision, a practice, or how people understand a problem. Utilisation-focused evaluation treats use by named intended users as the goal of the whole exercise (Patton, 2013).

  • Patton, M. Q. (2013). Utilization-focused evaluation (U-FE) checklist. The Evaluation Center, Western Michigan University. files.wmich.edu
  • Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., & Volkov, B. (2009). Research on evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30(3), 377–410. doi
  • Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56(3), 331–364. doi
Utilisation-focused evaluation

An approach that starts from the user: identify the primary intended users, design every step around their use, and report in a form they can act on (Patton, 2013).

  • Patton, M. Q. (2013). Utilization-focused evaluation (U-FE) checklist. The Evaluation Center, Western Michigan University. files.wmich.edu
  • Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Sage.

Nearby

The primer itself, and the ones either side

Read programme evaluation (opens in a new tab)Reading education research Education reform All 24 primers
Singapore