However, despite well-designed programs and dedicated staff, few organizations actually measure their impact. So how do they know what works and why? Could investments or programs be designed to be more effective? What could organizations do to make more money?
Andrew Dillon, a development economist and clinical associate professor at Kellogg, knows a thing or two about measuring social impact. This is because it helps researchers to Global Poverty Research Laboratory design studies that measure the impact of various poverty interventions around the world.
Dillon offers several suggestions for organizations looking to better understand and increase their impact on the communities they serve.
“Measuring impact is important not only for quantifying the return on a social investment,” says Dillon, “but it’s also important for cost efficiency and delivering impact at the lowest cost per beneficiary.”
You will need a “theory of change”
Whether you’re trying to make an impact halfway around the world or in your own backyard, you first need to understand the outcomes you hope to affect and how you plan to affect them.
The latter, Dillon says, is often neglected. “It’s not just about choosing metrics or choosing indicators or choosing some kind of variable to measure. Those things are important,” he says, but the real key “is connecting what the investment or program is to a ‘theory of change’ of how that investment or program is actually going to change the lives of beneficiaries ».
In other words, you must not only understand the end result but also how you get there: By what mechanism will your program or investment change someone’s life? What does this path look like?
Take, for example, a mentoring program that connects teenagers with working professionals, with the relatively specific goal of helping those teens achieve financial success. To design an effective program, you must first consider the mechanism by which your program helps adolescents. Will spending time with a mentor help them feel more informed and empowered about their career opportunities, leading them to pursue more lucrative options? Will the benefit come from accessing the mentor’s network? Or will it come from keeping teenagers more focused on school—or less on antisocial activities?
Each of these theories is plausible, but would lead to drastically different offerings. An empowerment- or access-oriented theory of change might include a job shadowing component, for example, while a problem-avoidance-oriented theory might come with safe after-school programming.
“Because mentoring programs can be designed in so many different ways, it’s really important to have a framework for measuring impact,” says Dillon.
If possible, do an experiment
Once you understand the theory of change, it’s time to determine if that theory is valid. This mentoring program actually Empowering teenagers to consider a more ambitious career path?
Whenever possible, Dillon recommends conducting a randomized controlled trial: that is, a study of the effectiveness of your program with both a treatment group participating in your program and a control group that is as similar as possible to the treatment group . (In a randomized controlled trial, this would be done by random assignment to either the treatment group or the control group.) If you don’t have the expertise to conduct your own trial, consider hiring an impact consultant.
A test like this will help you determine that any effect on your desired result is actually caused by your program, as opposed to some other factor. For example, if career-minded teenagers self-select into your mentoring program at higher rates than those who are less career-minded, then it will look like your program is highly successful, even if it does nothing absolutely.
With a well-designed experiment, you can even learn something unexpected. Dillon points to a recent study on the effects of workplace health interventions on worker productivity.
“It was known at the time that treating manual workers for malaria would reduce absenteeism and productivity of workers who were ill, but it was not expected that the health intervention could help even those workers who were not ill. with malaria at the time,” says Dillon. But when researchers conducted a large, randomized controlled trial, that’s exactly what they found.
“Ensuring testing and treatment for all workers allowed good workers to increase their effort and productivity at work because they didn’t have to worry about getting sick from the physical nature of their work. “Because they knew they weren’t sick, they could increase their effort,” Dillon says.
Before the study, the researchers’ theory of change did not account for how these interventions would affect employees well. The experiment allowed them to update their theory of change – and make a solid case for further investment.
If an experiment is too expensive, choose the next best thing
Not all organizations are able to conduct experiments (or even hire someone to do it for them). So the next best step is to consider whether there are existing studies that validate your theory of change. You may not have the resources to conduct an experiment on the effectiveness of your mentoring program, for example, but you should at least determine that there is evidence that your chosen mechanism—empowerment, access, problem avoidance—is connected by chance to the goals you have set.
Dillon points to the work of organizations such as the Poverty Research Lab and its partner organization, Action Innovations for Povertywhich conduct large-scale randomized controlled trials in a range of communities in order to determine precisely which interventions are effective in achieving outcomes through which mechanisms.
In addition, major foundations, donors, and even impact investment funds routinely assemble large evidence-based assessments—a trend that Dillon says is growing.
“Having well-designed studies that people can point to so they can clearly understand how program design features are linked to impact is very useful and a great public good, even if all parts of this study are not fully replicable in other communities,” says Dillon. “It allows us to validate our theories of change.”
Embrace the Process
Dillon acknowledges that historically, impact assessment has gotten a bad rap.
“A lot of evaluations in the past were by outside consultants, who would come in and just evaluate program models or evaluate the effectiveness of the organization, based on their own experience or their experience in other types of social impact spaces,” says Dillon. “You don’t necessarily want to let someone outside of your organization come in and just criticize you.”
But collecting data—data tied to a particular theory of change—evaluations can be a very positive experience, whether your analysis is conducted in-house or by a third party.
“In organizations that have a learning culture and want to learn how to improve, impact evaluations can be very well received because program designers and program staff have many innovative ideas about how to create impact and why things work.”
Even if you get the disappointing news that your intervention isn’t particularly effective, you can now try to determine what else you can do. If your nonprofit provides laptops to kids to help them learn, for example, but a lack of electricity or access to high-quality teachers prevents learning from happening, then you now have a good starting point for designing new interventions and a new trial.
“These ideas can really be tested and studied and tested and refined. And that can be really exciting,” says Dillon.
