Nishant Malhotra Founder of Middle Road OPC Pvt Ltd & The middle Road platform on the evolving Sustainable Finance sector

Ahoy Ahoy-Ahoy!

The middle Road

Welcome to the Kickass platform enabling social change & impact

Impact Evaluation Enabling Social Change & Impact

‘Theory of change is a dynamic, critical thinking process, it makes the initiative clear and transparent – it underpins strategic planning. It is developed in a participatory way over time, following a logical structure that is rigorous and specific, and that can meet a quality test by the stakeholder. The terminology is not important, it is about buying into the critical thinking.’ Helene Clark, ActKnowledge1

 

Impact Evaluation plays a fundamental role in understanding the impact of policies and interventions within the social and development sector. Today, the evidence-based policy approach is building up globally to drive in accountability, transparency, and learning. Data analysis facilitates the effectiveness of social impact and program evaluation of both experimental and non-experimental design-based approaches to solve complex problems in the international development sector. Theory of Change, a strategic decision-making tool forms the backbone of designing impact evaluation problems to address long-lasting long-term change. The educative tutorial herein discusses the underlying chain of preconditions/outcomes achieved through a series of interventions, measured through a set of indicators which are the underpinning of the theory of change model.

The underlying process of impact evaluation is aligned to the Theory of Change concept.

# Theory of Change 

Theory of Change (ToC) forms the bedrock of impact evaluation design. The impetus to the change framework came through the groundbreaking work by the Aspen Institute Round table on Community Change (Round table) titled The Community Builder’s Approach to Theory of Change: A Practical Guide to Theory Development for measuring and evaluating social interventions. Initially designed to facilitate a refresher course for building a theory of change model for community-based programs and initiatives, ToC philosophy permeates wide-ranging and long-lasting social change and impact within the social and development sector. Its footprints encompass impact-based giving to evidence-based analysis of policies and social interventions for a long-term wholesome change within our society. Theory of Change wears multiple hats; consider a deep-rooted philosophy, a strategic foundation to a change tool, its presence is paramount in today’s global ecosystem.

Over the years, the international development center went through a transformational change. Lack of transparency and accountability drove many actors away from the social sector.    Image left The middle Road

However, a rise in shareholder activism, backed by a collaborative model among various actors within the social and development ecosystem significantly the private sector and foundations kick-started strategic thinking for a sustained social impact. To enable a fundamental social change and impact into troubling and deep-rooted problems within the society, a scientific methodology and behavioral mindset change are essential for a prolonged social innovation. Impact Evaluation along with Theory of Change lays down a strategic shift in the philosophy among all actors within the social and business ecosystem enabling a quantifiable and persistent social change and impact on well-defined outcomes.

Take philanthropy for example. Thomas Tierney and Joel Fleishman in their breakout book, Give Smart, focus on the impact of Theory of Change fingerprints on the donor sector. Getting Clear and Getting Real became the core objectives in a three prolonged approach called Linked Learning. Developed by the James Irvine Foundation, Linked Learning changed the social paradigm within the educational sector in California especially among underserved students. The initiative focused on setting pathways to a long-tern outcome through a series of preconditions and outcomes achieved through a set of interventions. Interventions simplified are set of policy/policies implemented to nudge towards a long-term outcome among a sample population within a time period. This pathway of change serves as a backbone for impact evaluation.

“Linked Learning students accumulated 8.9 more credits by the end of high school.”

Linked Learning’s stupendous success in effectively motivating and preparing high school students for higher education and career progression epitomizes the confluence of an evidence-based approach with theory of change. Using the best available knowledge and data analysis, the team at James Irvine Foundation inferred the causality between that of student’s understanding of the importance of their learning to their future prospects as the root cause for motivating students for a better tomorrow. This fundamental truth defined the foundation’s career heavy pathway that has to deliver measurable social change and impact over the years for students, especially those from low-income families.

Linked Learning stupendous success in effectively motivating and preparing high school students for higher education and career ahead. Using the best available knowledge and data analysis, the team at James Irvine Foundation inferred the causality between that of student’s understanding of the importance of their learning to their future prospects of the root cause for motivating students for a more colourful future.  This fundamental truth defined the foundation’s career heavy pathway that has to deliver measurable social change and impact over the years for students, especially from low-income students. The students not only performed better in academics based on measures like higher average credits, better percentage points to graduate, less drop-out percentage points but also inculcated more self-belief, management, esteem along with better prospects of enrolling in college. The holistic development of students for a more enhanced and enduring future embodies the core philosophy of Impact Evaluation and Theory of Change.

# Defining the Objective of Change 

Any undertaking in impact evaluation begins with outlining a clear, specific, and concrete outcome that is both measurable and achievable. The first step in impact evaluation is also one of the most important since the pathways to change works backward. If the outcome is unrealistic, the whole exercise is futile. The journey beings with an objective which should be clear-cut in defining the stated long-term outcome clearly enunciating the budget, time horizon and measurable variables  for achieving the social change and impact. The long-term objective is very important to set up the type of experiment. Impact Evaluation process is divided into experimental and non-experimental design. Experimental designs are randomized control trials which set up before the experiment or process while non experimental designs like quasi experimental designs and regression analysis are conducted post experiment. The final end line survey and data collection comes out at the end stage of analysis and report generation in the process of impact evaluation. This further reinforces that importance of clearly defining the output for measuring the outcome setting future experiments for measuring impact.

Setting up a measurable goal aids in linking the early and intermediate preconditions together through key performance indicators making the next step of linking steps within pathways to change framework less tedious and more manageable. However, there are some caveats. First, is the outcome welcomed for the intended population served even if the objective is wholesome. Second, the importance of communicating the change to the intended population is so important to make them understand the intended social good. Research has found that a large number of deaths attributed to the outdoor population is due to the burning of biomass, driving the importance of cleaner stoves for indoor cooking. Indoor stoves burn a significant amount of biomass and yet studies have shown that there is no evidence for the demand for cleaner stoves among the population using indoor stoves. 4 The reason being that the population using indoor stoves cannot discern the link of causality between clear air and better health. Example India, indoor stoves are primarily used among poor households, especially in the rural sector. The underserved section of society usually gives low importance to health and physical wellbeing as a priority in life due to lack of education and social awareness. Therefore, it’s a must to communicate succinctly the importance of social intervention to the audience for their buy in for a smooth transition of the process. Finally, reinforce outcomes with logical assumptions. Assumptions need to be backed by factual data to lay down a sound foundation for a series of steps for a fruitful outcome. Assumptions based on artistic imaginations better be kept aside while designing experimental and quasi-experimental interventions.

# Pathway of Change 

The next stage includes setting up the experiment and requires partnerships across a wide range of  actors with different skill sets. This phase includes a collaborative, multifaceted approach with coordination among impact evaluation experts/consultants/investigators, monitoring and evaluation team who primarily design baseline and end line question set and interventions along with the business or social actors (non-profit, corporates, foundations, impact investors, etc). If the company has an in-house section for conducting, evaluating, and publishing the findings then you are a very different set of teams working on the problem-solving process. This part of the process is the most exhaustive part of the Theory of Change as it needs a series of preconditions to be synchronized together for the outcomes. This is where all the assumptions, drivers (external influences, regulatory or behavioural aspects) of the process comes into play.

Some organizations make external influences as their primary objective in enhancing wide-reaching social change and impact. Draper Richard Kaplan’s defines success as focusing on outstanding people to make a discernible social impact. Draper Richard Kaplan focuses on a comprehensive business plan with an eye on detail from its grantees.  After selecting the companies, it mentors, nutures and provides strategic guidance to its leaders. It lays down a road map for the social leaders and entrepreneurs through critical milestones.

John Wood, founder of the non-profit Room to Read is an example of an entrepreneur who has been able to scale his venture globally within a short period of time. 3 John left his cushy Microsoft job in China to start his social venture after a thought-provoking hiking trip in Nepal. Touched by on the face poverty in Nepal backed by a deep desire to educate improvised children especially girls, John’s journey in successfully establishing Room to Read is aptly captured in the book Leaving Microsoft to change the world. From its humble beginnings to supply books to underprivileged students in Nepal, Room to Read is a full-fledged global mission across countries that offers mentorship, scholarships, and equitable education to millions of low-income girls. To date, it has already impacted the lives of more than 18 million children worldwide.

 

# Measuring Impact

The pathway of change road map works backward taking the output into account and connects the model to the output through a series of intermediate steps. The process itself is an intervention that now breaks down into smaller interventions that address preconditions required for fulfilling the outcome. The change methodology especially works with minimal set of preconditions required to achievement the outcome. The output is mapped through a series of preconditions by the law of causality.

                                             If    X=====> Y

Surveys are one of the most vital tools that work as the medium of measuring indicators. Indicators are a good way of finding whether the correspondents have created or reached the precondition. 2  This sets the stage to ignore side effects of interventions which might be holistically good but not required for achieving the mission. Data analysis gives key insights in measuring the impact. The key innovation in understanding the measurement of impact lies in understanding the outcome of the counterfactual. Counterfactual outcomes are outcomes that would have happened if no intervention is implemented. To overcome the limitation of measuring both the outcome of the intervention at an intended time when the result is achieved with that of the counterfactual at the same time, experimental and quasi-experimental designs are regularly used for impact evaluation.

Randomized control trials are being increasingly used as a gold standard for measuring the difference between the evidence-based impact. In RCTs, two groups of treatment and control groups are selected randomly where the treatment group is given an intervention while the control group receives no intervention. The difference between the outcomes of both these two groups points to impact. The baseline survey is conducted on the sample population while end line survey results are taken after the completion of the experiment. Quasi-experimental designs use the difference in difference apart from techniques for measuring impact. Without going into technicalities, impact evaluation including RCTs are increasingly being  employed not only in the social sciences and development sector but also in the private sector for companies to understand the effect of promotions or new product launches on their target market segment to share an example.                  Image:  The middle Road

As we proceed on path of Impact Evaluation for enabling an overarching social change and impact within the world, evaluation tools become an essential tool to learn and understand impact analysis. Global turmoil due to inequitable resources and policies, political instability, civil wars, economic slowdown, and unequal economic rewards have complicated avenues of monetary help for underprivileged and underserved people around the world. As the development sector reels under the ongoing pandemic, the role of multiple sets of actors within the social sector is blurring with more need for capital from the private sector. In the present state of global flux, evidence-based measurements are exponentially implemented in the global ecosystem to not only understand the effect of policies on outcomes but also on how best we can manage the most optimal impact. Impact Evaluation has thrown evidence to the contrary as a measure of optimal impact. In 2016, Liberian government transferred reins of ninety-three government schools to a bunch of non-profits and for-profit enterprises. RCTs tests conducted threw a mixed and critical overview of impact.  Although results on an average were better, they came at an increased money spend per student, negating a clear and decisive outcome. 4

Mounting impacts results from international development sector clearly highlights that there is no single cure to tackling development problems whether it is alleviating poverty to equitable education for all. Different sets of problems need customised solutions. Although data analysis is playing a significant role in the development ecosystem, it is not a substitute for human touch. Instead of making cosmetic changes for the betterment of people, impact evaluation with the embedded theory of change framework lays down a paradigm for change not only within the external environment but also mindful changes within ourselves.

References

  1. Review of the use of ‘Theory of Change’ in international development Isabel Vogel for the UK Department of International Development
  2. The community builder’s approach to Theory of Change: A practical guide to theory development
  3. Give Smart: Thomas Tierney and Joel Fleishman
  4. Good Economics for Hard Times Abhijit Banerjee and Esther Duflo
  5. Step by step guide to impact evaluation
  6. Roomtoread.org
  7. Asian Development Bank

 

 

 

 

 

 

 

This website uses cookies to ensure you get the best experience on our website.

%d bloggers like this: