18-05-2012, 01:22 PM
Realist Evaluation
2011_WP_RealistEvaluationSeminar_CecileKusters.pdf (Size: 472.15 KB / Downloads: 100)
What is Realist Evaluation?
The term ‘realist evaluation’ is drawn from Pawson and Tilley’s seminal work, Realistic Evaluation (1997). It is an approach grounded in realism, a school of philosophy which asserts that both the material and the social worlds are ‘real’ and can have real effects; and that it is possible to work towards a closer understanding of what causes change. The word ‘Realistic’ in the book title is a play on words: “real” because it refers to work done in the real world; realist because it is grounded in realism; and “realistic” because it is a form of applied research, which is “pursued to inform the thinking of policy makers, practitioners, program participants and the public”.
Realist Evaluation changes the basic evaluation question. It does not ask what works, or does this work. It asks ‘what works for whom, in what contexts, in what respects and how'. A realist approach assumes that programmes are ‘theories incarnate’. That is, whenever a program is implemented, it is testing a theory about what ‘might cause change’, even though that theory may not be explicit. One of the tasks of realist evaluation is therefore to make the theories within a programme explicit, by developing clear hypotheses about how, and for whom, programmes might ‘work’. The implementation of the programme, and the evaluation of it, then tests those hypotheses. This means collecting data, not just about programme impacts, or the process of programme implementation, but also about the specific aspects of programme context that might impact on programme and about the specific mechanisms that might be creating change.
Realist Evaluation in the broader context of theory-based evaluation
What does the word ‘theory’ mean in theory-based evaluation? Four kinds of theory can be distinguished: philosophical theory, evaluation theory, programme theory and substantive theory. The deepest level, philosophy, deals with assumptions about the nature of the world and how we understand it. These can never be proven. Philosophy includes theories about ‘what exists’ (ontology), and what can be known about it (epistemology). These theories or beliefs influence all the other kinds of theory. Evaluation theory deals with how things can be evaluated and what can be known about the results. For example, some people argue that different philosophies underpin the quantitative and qualitative paradigms in evaluation. Programme theory describes the theory built into every programme. There are different ways of representing program theory, including programme logic and theory of change. Substantive theory operates within a particular domain or discipline, for example psychology.
Realist Evaluation can be represented at all four levels. On the level of philosophy, Realist Evaluation is grounded in realism and in systems theory (for human services programs, a particular kind of systems theory, called complexity theory, is often useful.) Realist evaluation itself is a particular evaluation theory. Realist Evaluation develops a particular kind of programme theory, structured as Context- Mechanism-Outcome Configuration (CMOCs). Lastly, substantive theory feeds realist evaluation with clues on the mechanisms (e.g. the combination of reasoning and resources that enable a programme to ‘work’) through which programs work and the contexts in which they will work.
Assumptions about knowledge and its meaning for evaluation
Experiences and understandings are filtered through the human brain and language. Every human brain is different and interprets differently. Language also shapes how a person interprets his experience. Some people argue, therefore, that it is impossible to state that one person is right and another wrong. However, realists believe that the real world ‘regulates’ and sets boundaries on what is reasonable to believe, and that knowledge can be improved over time.
With regard to evaluation, this means that there is no absolute certainty about evidence to date, and also no absolute certainty about evaluation findings. Evidence is just ‘the best we think we know so far’. Evaluations should therefore be iterative and knowledge should be treated as cumulative, across policies and programmes. Realist evaluation is designed to enable improving understanding over time: how can learning from one programme be taken to another?