Understanding Counterfactual

Impact Evaluation of Development Interventions A Practical Guide Howard White David A. Raitzer Asian Development Bank is a highly recommended paper to understand the methodologies used in impact evaluation. Asian Development Bank is a highly reputed multilateral bank with a critical focus in selected Asian markets and an essential reference for this educational series on Impact Evaluation. Theory of Change is the underlying program logic on which projects and interventions are designed –Kindly refer to Community Builder’s Approach to Theory of Change: A Practical Guide to Theory Development and The middle Road with Dr. Helene Clark, a foremost expert on this framework.

The podcast with Dr. Helene is a must for this course, as quiz questions will be based on this podcast. Nishant Malhotra also interviewed Dr. Helene for promoting the Theory of Change course offered by Helene’s on profit, the first paid promotion from the thought leader platform. Impact evaluation enabling social change and impact published under Insights is required part of the read. The first step to understanding Impact Evaluation is to measure the effect of policies or projects.

Statistics play a crucial role in quantifying impact, yet there are many times it’s not easy to state key performance indicators for measuring the intended outcome. For example, you want to measure the development of students’ personalities and come out with various interventions to achieve the goal. It will not be easy to quantify the improvement of the target audience’s personalities though you can develop a framework on specific qualitative characteristics rating them for a final score. At the same time, many outcomes are easy to measure. One could be the climate action achievement in a given geographic area. You want to measure if the policies implemented have caused a reduction in pollution within a specific period. A reduction in greenhouse gas emissions doesn’t imply that it’s because of the policies implemented. A drop in pollution could result from factors resulting from the policies implemented or independent of the policies implemented. With this, we come to the question of causality, a key measure for understanding impact evaluation. Interventions are frequently used term within the development sector. Interventions mean projects or policies designed to achieve either a series of casually interlinked outputs to achieve the long-term outcome.

Projects in the social sector use the strategic framework of the Theory of Change framework that looks at the outcome change and builds a series of casually related medium-term interventions to achieve the outcome. For example, the outcome you want to achieve is a greener environment. Output can increase the reliance on alternative fuel as a percent of energy consumption over time. This output is an enabler for the outcome.

What is a Counterfactual ?

Image Reference Source: Impact Evaluation of Development Interventions A Practical Guide Howard White David A. Raitzer; Asian Development Bank

Impact = The reduction in CO2 emissions in areas where the policy is implemented at time t+1 – The reduction in CO2 emissions in areas where the policy is not implemented at time t+1 

For the above experiment, keep other factors constant that lead to a reduction or increase in CO2 emissions. A higher number of installed mini-grids should lead to an enormous impact.

Note: Here, we assume that the installation of mini-grids leads to a reduction in CO2 – an example of World Bank statistics. 

Remember Back to the Future. Michael J Fox goes back in time to change the future. The impact of the time machine is the difference between the scenario from where Michael goes back in time to that of the one he returns in the second part.

Comparison with the counterfactual is the actual impact of any intervention.  

Impact Evaluation Example 

An excellent tool to understand the impact of interventions would be to compare two outcomes – one when there is no intervention given to one where the intervention is provided. However, the problem is how to compare the two outcomes simultaneously as both cannot coexist. Randomized Controlled Trials (RCTs) are precise tools that try to create two groups, one wherein an intervention is delivered to another where no intervention is administered— the difference measures the impact of the intervention. RCTs are not easy to implement and explain; a separate module will discuss this technique.  What would have happened if there was no intervention compared with what would have happened if there was an intervention. Example government wants to reduce pollution, focuses on clean energy. World Bank estimates 210,000 mini-grids powered by solar energy would help avoid 1.5 billion tons of CO2 emissions globally. Based on a comprehensive study taken by The World Bank, the mini-grids installation will play a significant role in achieving universal access to electricity by 2030; –  40 percent of all installed capacity will come from mini-grids with the potential to provide electricity to as many as 500 million people by 2030. Therefore, the government implements scaling mini grid project with IFC.  A policy that fosters installation of mini grids in many rural and semi urban locations. Let this be the output you are mapping. How will you know the true impact of this policy?

In the above figure, if at time t the policy is implemented, time t+1 measure an increase in the number of mini grids installed to gauge the effect of the policy on the sample population.

The impact is the reduction of CO2 emissions in localities, locations, cities, and villages where the policy is implemented to decrease in CO2 in localities/areas where the procedure is not implemented. The group sample could be a locality, city, village, etc. The two groups have to be similar to compare with no selection bias etc. For now, assume all other factors are constant; the difference in impact — between factual output with counterfactual output. Since both factual and counterfactual cannot exist simultaneously, specialized techniques like RCTs are designed.

A city builds a 4 lane highway between the city center and a bustlingly suburb about 10 miles away. The local government wants to conduct a impact evaluation on the project. 

Measure the impact on the economic development of the local community, on the business activity due better infrastructure and mode of transportation

Was there a reduction of carbon emissions due to the increase in the efficiency of transportation?

Did building the highway achieve the stated goals laid out in national policy statement?

The next lesson will talk about the Theory of  Change framework.