This article puts institutional economics concepts to work to help identify the proper role of evaluation in organizations and circumvent key obstacles to evaluation use. Looking at the role of evaluation in bureaucracies through an economics lens has its limitations. But addressing ‘the rules of the evaluation game’ helps to complement currently dominant approaches that concentrate on evaluation quality and practices. In concert with systems thinking, the neo-institutional economics perspective provides useful pointers for the design of evaluation governance configurations geared to organizational learning and accountability.
Commissioners of impact evaluation often place great emphasis on assessing the contribution made by a particular intervention in achieving one or more outcomes, commonly referred to as a ‘contribution claim’. Current theory-based approaches fail to provide evaluators with guidance on how to collect data and assess how strongly or weakly such data support contribution claims. This article presents a rigorous quali-quantitative approach to establish the validity of contribution claims in impact evaluation, with explicit criteria to guide evaluators in data collection and in measuring confidence in their findings. Coined as ‘Contribution Tracing’, the approach is inspired by the principles of Process Tracing and Bayesian Updating, and attempts to make these accessible, relevant and applicable by evaluators. The Contribution Tracing approach, aided by a symbolic ‘contribution trial’, adds value to impact evaluation theory-based approaches by: reducing confirmation bias; improving the conceptual clarity and precision of theories of change; providing more transparency and predictability to data-collection efforts; and ultimately increasing the internal validity and credibility of evaluation findings, namely of qualitative statements. The approach is demonstrated in the impact evaluation of the Universal Health Care campaign, an advocacy campaign aimed at influencing health policy in Ghana.
To be successfully and sustainably adopted, policy-makers, service managers and practitioners want public programmes to be affordable and cost-effective, as well as effective. While the realist evaluation question is often summarised as what works for whom, under what circumstances, we believe the approach can be as salient to answering questions about resource use, costs and cost-effectiveness – the traditional domain of economic evaluation methods. This paper first describes the key similarities and differences between economic evaluation and realist evaluation. It summarises what health economists see as the challenges of evaluating complex interventions, and their suggested solutions. We then use examples of programme theory from a recent realist review of shared care for chronic conditions to illustrate two ways in which realist evaluations might better capture the resource requirements and resource consequences of programmes, and thereby produce explanations of how they are linked to outcomes (i.e. explanations of cost-effectiveness).
The integration of realist evaluation principles within randomised controlled trials (‘realist RCTs’) enables evaluations of complex interventions to answer questions about what works, for whom and under what circumstances. This allows evaluators to better develop and refine mid-level programme theories. However, this is only one phase in the process of developing and evaluating complex interventions. We describe and exemplify how social scientists can integrate realist principles across all phases of the Medical Research Council framework. Intervention development, modelling, and feasibility and pilot studies need to theorise the contextual conditions necessary for intervention mechanisms to be activated. Where interventions are scaled up and translated into routine practice, realist principles also have much to offer in facilitating knowledge about longer-term sustainability, benefits and harms. Integrating a realist approach across all phases of complex intervention science is vital for considering the feasibility and likely effects of interventions for different localities and population subgroups.
Scientific evaluation seeks to develop and test theories that describe and explain the value of interventions into the world. Realist approaches to scientific evaluation tend to be strong on theory and explanation, but lack adequate tests or means of validating theory. The focus of this article is the potential for randomisation and experimentation to provide evidence for transfactual (i.e. reusable or portable) context–mechanism–outcome configurations (CMOs) in complex adaptive systems. The article is not concerned with attribution of outcomes to past programs but for developing scientific knowledge that can be used for future interventions. It seeks to elucidate the warrant that underpins the randomised controlled trial (RCT) and why it is useful in some fields of science but less so in complex social systems. Realist RCTs are considered but rejected; instead a form of propensity score matching is proposed for testing realist program theory, estimating the effect size of a purported CMO, and generating scientific knowledge for developing more effective interventions into complex social systems.
Pawson and Tilley suggest a unique strategy for conducting qualitative interviews within realist evaluation studies. Theories are placed before the interviewee for them to comment on with a view to providing refinement. The subject matter of the interview is the researcher’s theory and interviewees confirm, falsify, and refine this theory. This relationship – described as a teacher–learner cycle – is integral to realist evaluations. This article provides an overview of how interview techniques have been applied in realist evaluations in the last decade as well as suggesting two guiding principles. The first one relates to the design of realist studies and the second one explains how to ask questions like a realist, and proposes three different phases in realist interviews: theory gleaning, theory refining and theory consolidation. The article aims to contribute to a growing understanding of the practical and epistemological challenges presented by primary data collection in realist evaluation.