By the end of this unit, participants will be able to:

• Recognize the different types of evaluation used is organisational settings and when and how to deploy them to capture the impact of your work.
• Discern the standard elements of a theory of change.

Background

Change efforts that manage to gather resources and traction have a better chance at success. Demonstrating the value of an idea or a project requires the ability to explain how change will happen as a result. Concretely, this means having an account of the way in which outcomes will be assessed.

Unit 8 is about evaluation: how to develop a theory of change, the purpose of different types of evaluation and when to use which.

Theory of Change (or ‘ToC’) is a tool used widely to support the design, development, implementation and evaluation of socially innovative work. While it does not exhaust the range of tools that are available when it comes to impact assessment, a theory of change helps with clarifying intended outcomes, describing the change and the action that will in theory produce outcomes and, ultimately, with determining if the changes proposed had the effects intended.

Core Concepts

Emergence

Emergence (see Unit 6) happens when efforts that have been happening locally, or at the scale of niches (see Unit 4), start to interact with each other to cause change that goes beyond their individual impact. Emergent change in a system may be in part intentional, or at least aspirational. But it is difficult to predict and to control and it cannot be reduced to the actions of the parts of the system.

“In nature, change never happens as a result of top-down, preconceived strategic plans, or from the mandate of any single individual or boss. Change begins as local actions spring up simultaneously in many different areas. If these changes remain disconnected, nothing happens beyond each locale. However, when they become connected, local actions can emerge as a powerful system with influence at a more global or comprehensive level.” (Weathley and Frieze)

Example: Murmuration

Watch here.

 

A murmuration is a sort of shapeshifting, morphing swarms of starlings. A murmuration is an emergent phenomenon: each of the movements of a murmuration look coordinated, but the it is the results of each bird following, at an individual or local level, a set of simple rules.

Figure 8.1. Murmuration: courtesy of Thrumyeye: https://www.deviantart.com/thrumyeye/art/Starling-Murmuration-883132099.

Individual birds monitor specific aspects of their movements and the movements of birds around them with the purpose of implementing three simple rules we could formulate as follows:

• Attraction: if you are in a bird’s “attraction zone”, move toward them.
• Repulsion: move away from a bird’s “repulsion zone”, otherwise you both fall.
• Angular alignment: align yourself to your neighbour so that you loosely follow direction.

What makes a murmuration an emergent phenomenon is the fact that it is more than a collection of individual little black birds following attraction, repulsion and angular alignment rules: as a murmuration, the collection of individual birds transform into mesmerizing configuration that may count hundreds of thousands. They are a breath-taking wonder, a pulsating, swooping, living, harmonized whole.

The murmuration is an allegory for systems thinking. Systems can achieve things that are greater than the sum of what their parts can achieve. Likewise, people, organisations and institutions can aspire to create emergent, systemic change.

However, individually, they only have agency over specific elements (e.g. projects, organisations) in their own ecosystem, namely those within their network of influence. One important aspect of systemic change, then, consists in having the tools to design, test and effect purposeful systemic change, i.e. to act steer a systems as a whole. One crucial aspect of success in contexts of emergence is to know how to track and assess intentional systemic impact. In the context of systems transformation, those involved in steering change also need to be able to track transformation that emerge as a result of their local action, i.e. at organisational and institutional scale. This is vital because changes that emerge from combined, collective efforts in an ecosystem may only create windows of opportunity, and it is important to be in a position to recognise and seize them before they close.

In “Using Emergence to take Social Innovations to Scale,” Margaret Wheatley and Deborah Frieze argue that, to enable systems change, fostering connections between networks takes precedence over developing social entrepreneurs individually. But it is reasonable to assume that both can and should be done, for optimal results.

The Strategic Planning Fallacy

In The Fall and Rise of Strategic Planning, Henry Mintzberg warns against the fallacy of assuming that strategic planning (a widespread approach to organizational change and impact) always results in strategy making. Strategic planning was initially introduced in business theory in the mid-1960’s in the Unites States and was quickly embraced by corporate leaders as the single best way to design and implement organisational strategies. Minzberg points to the fact that thinking about strategy, i.e. delivering a set of step-by-step instructions to implement an idea does not always result in doing things strategically and actually implementing that idea.

The strategic planning fallacy occurs when strategic planning and strategy making are conflated, i.e. when they are not sufficiently differentiated. It rests on three mistaken assumptions:

The most successful strategies for systems thinkers are visions that articulate aspirations for the future: they are closer to foresight (the study of futures) than to plans.

Deliberate Strategy vs Emergent Strategy

Organisations need to be methodical about planning. They need to set intentions, plan a course of action, determine what they need to get there and what actions they need to take. Being deliberate is good practice and there is a role for quantitative data and formal analyses in strategic thinking because they can help understand aspects of situations on which we are attempting to effect change from a different perspective. In this sense, strategic thinking can help expand the consideration of issues in a team setting.

But strategic thinking is not, all on its own, strategy. More importantly, it is not emergent strategy. In the context of systems work, including more traditional approaches to strategic thinking, strategy-making should be consistent with leaving room for emergence. The creation of an emergent strategy is an exercise that is deeply reflexive and consists in understanding how different actions taken one at a time by an organisation may converge into patterns that may not always be predictable and thus require the type of flexibility that would be appropriate in a context of emergence.

In the social and public sectors, recognizing that control is an illusion is imperative: the best laid plans will go sideways! One way to mitigate the consequences is to foster agility, which is the ability to change course in the middle of a plan, if necessary. Learning when to let go and move in a new direction is a key competence. Likewise, it is crucial to be ready to take advantage of what emerges, rather than hold tight to what has been planned.

Being at once both deliberate and agile makes it possible to maintain a course, while remaining open to emergent phenomena.

Consider again the case study of the Registered Disability Savings Plan (Unit 6). Etmanski and Cammack’s window of opportunity came at a time when they experienced their greatest sense of failure. They had thrown their hopes into a Liberal government winning the federal election. They had seen the RDSP posted on the Liberal platform. They thought they had paired with a winner, but the Liberals lost and they assumed they had failed. In the end, it was an old acquaintance – a Conservative Party member- they had met many years earlier who finally brought their project to the attention of the Minister of Finance and put the RDSP back on the path to success.

As social innovations are developed, adapted and adapted again, opportunities present themselves that can change their trajectory. There is no guarantee. And one must stay alert to the possibilities. There is no way to know in advance when or where a window of opportunity will open, but one can be prepared and one needs to track the progress along the way.

The Role of Evaluation

When it comes to deploying or implementing a strategy, evaluation matters for several reasons:

• Funders typically expect to be presented with evidence of progress and impact.
• Community members and other stakeholders may be interested to know how programs are being received.
• Board of Directors may need the information to determine how to better support the executive team.

It is crucial to reconcile the various roles evaluation might play in social innovation with the fact that, as an emergent phenomenon, change can occur asynchronously and outside of a plan or timeline. What this means is that approaches to evaluation around social innovation needs to accommodate the possibility that what was planned needs to be adjusted in the face of emergent opportunity. This creates a tension between the need to adopt data-driven (formative, summative) evaluation methods and the possibility that the outcome be different from the one that was planned. For this reason, we need an approach to evaluation that favours emergence over rigid logic models.

Different types of evaluation have different purposes and may be better suited to specific phases of a system’s adaptive cycle. Based on historical trends in program evaluation, VeLure Roholt et al (2023) propose a relational approach, the “learning partnership” model as an alternative. Generally speaking, all approaches to assessment are deployed to help determine whether and how a project or organisation is having impact.

Developmental Evaluation

[See this video and this resource from Better Evaluation]

 

“Developmental evaluation applies to an ongoing process of innovation in which both the path and the destination are evolving. […] Where more traditional approaches to evaluation try to predict the outcomes of the innovation and focus measurement on those goals, developmental evaluation is intended to support innovation within a context of uncertainty.”

– A Developmental Evaluation Primer, by Jamie A.A. Gamble with The J.W. McConnell Family Foundation

A developmental evaluation is an approach that is particularly well-suited to:

• The early stages of social innovation.
• Radical program redesign.
• Replication.
• Complex issues and crises.

In these contexts, developmental evaluation can help support the processes we associate with defining the problem, framing concepts/questions, testing quick iterations of an idea as well as with tracking developments and surfacing issues.

Developmental evaluation supports the process of innovation by enabling exploration and experimentation, which in turns can help with sense-making, i.e. it can help those involved to create an honest and elegant narrative around the project. A developmental evaluation process aims to help the group interpret, explain and expound on their explorations by providing continuous input and intelligence around the emerging processes, decisions, events and relationships, support synthesis and feedback and identify evidence gaps where possible that may require additional research.

In the exploration/reorganization phase of a system (Unit 3) or the initial design of an idea, those in innovation roles will need to work with funders and team members to create shared frame of reference and understanding. In the exploration/reorganisation phase, social innovation can leverage developmental evaluation to articulate a narrative around the a project’s deployment.

Formative Evaluation

Formative evaluation approaches – keep in mind that there are several different kinds – may be best suited to the exploitation phase of a system’s adaptive cycle, or the point at which a project’s strategy has been identified and a path forward has been chosen (Unit 3). In the exploitation phase, innovation stakeholders need to gather evidence that the initiative is achieving its purpose. Formative evaluation is setup at the beginning of a project’s execution and used to get baseline data that can be compared against data gathered post-execution.

A formative evaluation can help an organisation determine whether they need to pivot if things aren’t working, or it can help identify opportunities to double down in areas of effectiveness. It helps understand how inputs and outputs are contributing to overall objectives. It can also inform how a program should be designed/redesigned before launching in full. Formative evaluation is often identified in the form of pre-/post-program surveys – to gather a baseline from which to determine direction and to contrast in the summative phase.

Summative Evaluation

Summative Evaluation is possibly the better-known approach to assessment. Everyone has been asked to complete a survey, for example, to provide feedback after they have taken part in an event or a program.

Summative evaluation is an approach to program or project assessment where the focus is on the outcomes. Typically, it is used in combination with formative evaluation approaches. For example, a program team may run a survey of clients/users of a program to gather evidence like:

• How did they enjoy the program?
• What aspects were effective?
• What aspects could be done better?

In academic contexts, tests, final exams, etc. are also types of summative evaluation. A grade is given that is intended to reflect a student’s ability to give evidence of their learning at the end of a course or program. Likewise, when students are asked to provide their feedback on a course, this is usually for the purpose of determining some measure of instructional success. By contrast, an instructor who would conduct a “knowledge check” via class discussion, for example, to gauge students’ prior knowledge would be relying on formative, not summative evaluation.

Outputs and Outcomes

In the context of evaluation practices, the distinction between outputs and outcomes is a crucial one.

Outputs are “countables”. They include, for instance:

• Survey answers.
• Sales/Subscriptions.
• Website traffic.
• Registrations/Participation numbers.

Through its outputs, an initiative may generate change. An initiative’s outcomes correspond to the results or change generated through that initiative’s outputs.

The number of people who participated in a program is an output of that program, but the growth or learning people experienced as a result of their participation is the program’s outcome. The impact of that program might be some cultural shift at a broader scale and outside the immediate sphere of influence of the project. The terms ‘outcomes’ and ‘impact’ are sometimes used interchangeably. However, their meaning is different: generally speaking, outcomes are understood to happen on the short to medium term and impact is considered to evolve over longer timelines.

Because impact is remote and somewhat unpredictable, conventional wisdom is to focus on outcomes. However, outcomes too are difficult to measure and hard to assess. One way to assess outcomes is to ask participants in a program to fill out entry and exit surveys (pre- and post-tests). The consensus is that longitudinal studies are better, although far more resource-intensive.

Work in the social sector, especially in non-profit, often requires that organizations account for a program’s outcomes. Foundations and government funding agencies expect evidence of impact to justify investment. Outcomes take time to achieve, but program funding is often only available for a short period of time. Organisations may be limited in their ability to quantify outputs, but they generally have far more trouble articulating outcomes and impact.

One alternative to standard evaluation paradigms, and one that is especially apt for systems mediation is to adopt an approach based on a theory of change. In a supportive funding environment, non-profits can better assess the true impact of their work by taking the time needed to establish and develop an understanding of what happens over time to benefit their clients/users in a context of emergence.

Theory of Change

We have seen in the previous sections that strategic planning arguably has a limited utility in complex ecosystems. Articulating a pathway to change and improvement is nonetheless imperative. A theory of change can help bypass some of the limitation of standard strategic tools to create cohesion around a project and get partners and team aligned on the objectives as well as on fundraising/financing, supporters, resources, collaborators and more. A theory of change also supposes that harvesting knowledge for evaluation purposes is an opportunity to engage in a reflection that can feed back into decision-making about new directions and designs.

At a very high level, a theory of change describes the what, the why and the how of a project. As such, a theory of change integrates all three types of evaluation: developmental, formative and summative. Broadly speaking, at the beginning of a project/program, the definition of the intended impact of an initiative or project is best supported by developmental evaluation. Setting out the activities and outputs is a matter of formative evaluation, while outcomes pertain to summative evaluation. Outputs and outcomes are components of any theory of change. Outputs and outcomes must be identified and then measured throughout execution, which can help an organization recognise when to move forward or how to respond to a particular outcome.

As such, a theory of change is a description of why a particular initiative will produce results: it articulates how change happens in the short, medium and long term to achieve the intended impact. It can be represented in a visual diagram, as a narrative, or both.

A good theory of change:

• Explains how the activities undertaken produce series of outputs that are likely lead to the intended or observed outcomes and impact.
• Helps develop better strategic evaluation questions, identify key indicators for monitoring progress, ascertain gaps in available data, prioritize additional data collection and provide a structure for data analysis and reporting.
• Combines existing evidence and vision.
• Makes explicit the appropriateness of the underlying assumption about change, approaches and action.

Innovation actors need to be ready to take advantage of what has emerged. The ability to embrace emergence consists in navigating the balance between maintaining a course of action that is good enough while taking advantage of opportunities as they appear.

Being deliberate about planning – setting intention, planning a course, determining what is needed to get there, and what actions to take – is good practice. But recognizing that only so much can be controlled in human systems is also important.

Data from evaluation is needed to let funders know about the progress an initiative is making, inform community members about the way your programs are being experienced, and can help a board of directors better support members of the team.

Developmental evaluation is particularly suited in the early stages of social innovation, radical program redesign, complex issues, and crises. It helps with framing concepts, the rapid testing of iterations, and with tracking developments and surfacing issues.

Developmental evaluation is a process that allows innovation actors to interpret intent and to explain and explore a course in order to pursue it. It also serves to articulate simple, elegant, straightforward and honest narratives to make sense of exploration and experimentation.

Formative evaluation is useful to determined if things aren’t working and an initiative needs to pivot, or to identify areas of effectiveness in which additional investment would be judicious. It may take the form of pre-program surveys, to establish a baseline. In a formative evaluation, data is gathered to assess the effectiveness of a initiative and guide decisions on what to adopt and how to improve it.

A summative evaluation is an assessment that focuses on outcomes and usually requires the priori definition of a set of criteria or goals.

In the context of evaluation practices, the distinction between outputs and outcome is important. Outputs are “countables” that can be determined on the basis of survey answers, sales statistics, website traffic, or participants numbers. Outcomes and impact are the changes the program is expected to create, and can be determined on the of data collected, for instance, through self-assessment or focus groups.

A theory of change is a representation the way in which a particular initiative will be effective, and is designed to describe how change is meant to happen in the short, medium and long term in order to achieve the intended impact. It can take the form of a visual diagram, a narrative, or both. A theory of change will typically makes use of all three kinds of evaluations, and include projections of outputs and outcomes.

Slides

Partner Engagement

Assignment

References