Designing and Evaluating for Change
Many organizations claim to “make a difference” in the world. And many do—by providing supplies to those in crisis, making healthcare more accessible, expanding education opportunities, training farmers in new agriculture methods, the list goes on. Hundreds of millions of people—even billions—see huge differences in their lives thanks to these services.
But what does it mean to really make a difference? And for programs that claim results across the entire community or country, how do you measure impact on that sort of scale? As a graduate student studying international development at Georgetown University, I’ve tried to better define what success looks like in international development. More and more frequently, donors, governments, and those we work with are demanding evidence that their investments of time and money are proving effective. Our challenge, and the challenge of any social entrepreneur, is to find ways to collect and demonstrate that evidence.
Finding Impact in Complex Situations
Sometimes the question, “What is the impact of this program?” is more complicated than it appears. This is particularly true in programs that have atypical goals and methods.
For example, Diseña el Cambio (Design the Change) is a Mexican nonprofit that is redefining what it means to “make a difference.” Diseña el Cambio doesn’t provide services directly to those in need; instead, it empowers children to transform their own communities. Children identify what could be improved in their communities, and then gather the support of the rest of the community to make it happen. Over the last year, Diseña el Cambio supported over 8,000 student projects ranging from creating a children’s corner in the library to cleaning up trash around a local well. The process pushes communities to recognize the idealistic visions of their children, and then to work to make that vision become reality. The Diseña el Cambio approach is based around four stages: feel, imagine, do, and share.
As consultants in YouthActionNet’s University Connect at Georgetown program, my classmate and I wanted to work with Diseña el Cambio because they were passionate about both creating and measuring impact, but lacked the knowledge on how to collect that evidence. We agreed to create a more concrete plan for evaluating Diseña el Cambio’s impact to help them address the rigorous evidence expectations of modern donors.
We saw firsthand the challenges with evaluating impact in sectors where the results are not clearly defined. The “soft” skills of leadership, self-esteem, and trust in community that are developed by the program activities are hard to measure accurately. And because every project is different, determining the overall social impact of Diseña el Cambio’s work is nearly impossible.
Social versus Individual Impact
The development community has taken some big steps in being able to identify individual improvements in self-esteem, confidence, and optimism. The more difficult challenge is how to evaluate those programs that seek impact on a broader scale.
One of Diseña el Cambio’s projects in the last year was a collaboration with six schools which built strong partnerships with government, businesses, parents, etc. Though the specific projects they completed were excellent, those impacts were minimal compared to the potential of the new connection between so many groups in the community. This partnership laid the foundation for future interactions, and for a society that is much more united in identifying and addressing problems. But how can organizations like Diseña el Cambio measure that transformation over time?
Three Steps for Evaluation
We recommended Diseña el Cambio take the following three steps to prepare its evidence base:
Clarify the theory of change. How do you define success? How far do you see the effects of your program spreading (to individuals only, or to communities)? What does success look like at each of those levels?
- Set specific indicators for success. After defining success, think carefully about how elements of that success can be measured. Is the information readily available, or will you need to conduct surveys of participants?
- Develop an evaluation culture. To meet the world’s demand for evidence, development organizations should allocate sufficient financial and staff resources to make evaluation a priority.
The development community has a long way to go in being able to support the types of innovative work that really makes a difference. Diseña el Cambio is not alone in trying to clarify its impact, or in its struggles to define the measurable elements of both individual and social impact. The evaluation process will become more complicated as development projects extend their reach to increasingly remote spheres.
I want to be part of the exploration of development’s new frontier—finding better methods to evaluate social and community-wide impacts. The University Connect project helped me get started by showing me the complexity of evaluation in practice. Ultimately, advancements in this space will require the dedicated efforts of organizations that are committed to defining their goals and proving their success.
Sierra Frischknecht is a master’s student at Georgetown University’s Global Human Development Program, where she is focusing on evaluation. Previously, Sierra worked with Mercy Corps, where she developed a toolkit to train Mercy Corps’ 40+ field teams in local advocacy strategies. She holds a BS in Business Management from Brigham Young University and lives in Arlington, Virginia with her one-year-old son and husband.
Learn more about the 2015 University Connect at Georgetown practicum.