For providers to deliver the best possible services to newcomers, they must be able to measure the success of past and ongoing projects—with a basis in concrete, rigorous data. This blog post, written by Switchboard’s Monitoring and Evaluation (M&E) team, will introduce you to some best practices for measuring success: developing a theory of change, defining effective indicators, planning for data collection, and evaluating your project.
Introduction
Measuring success involves collecting and analyzing data to assess how well a project has achieved its stated goals. This helps service providers change and improve programs to better support newcomer clients. By measuring successes, as well as areas for improvement, project staff are also committing to accountability with key stakeholders, especially clients.
1. Start with a Strong Theory of Change
Measuring project success starts with a strong theory of change. A theory of change is a type of logic model that defines all the building blocks required to bring about a given long-term goal by illustrating the causal pathways between activities, outputs, outcomes, and objectives. It is best practice to create theories of change collaboratively with other team members during the program design phase. However, they can also be developed for existing programs as part of, for example, strategic planning or grant renewals. Explore Switchboard’s resources for a detailed description of how to develop a theory of change.
2. Develop Effective Indicators
Once your project has a strong theory of change, you will need to develop effective indicators: that is, variables and factors that embody and reflect the changes associated with an intervention.
Choose your indicators carefully: they should be meaningful, serve a programmatic purpose, and enable you to capture all data required by funders. For example, in an employment project, tracking the number of clients who achieved an income that is sufficient to meet their basic needs could provide insight into whether clients are being connected to the right employment opportunities. This data can also be reported to funders to demonstrate the employment project’s success.
Switchboard offers guidance on how to determine if indicators for outputs, outcomes, and objectives are purposeful, as well as a checklist to help build SMART indicators.
3. Plan Your Data Collection
To reliably measure your project’s success, selecting appropriate indicators is only one step. You also need to plan for how you will collect and weight your data.
Commonly used data collection methods include surveys, focus groups, observation, case studies, review of existing materials like project documents, or interviews. These methods can be used separately or together.
Consider some of the following strategies as you create your data collection plan:
- Develop an indicator matrix to map out exactly how each indicator will be calculated for each project objective, outcome, or output. This includes listing the necessary data points, data sources, targets, and responsible individuals. Switchboard offers a downloadable and fillable template to help you compile your own matrix.
- Set up systems for data collection, analysis, and reporting using an M&E plan. The M&E plan also includes the theory of change, logframe, and indicator matrix. Switchboard offers an M&E Plan Template and accompanying checklist which will help you formulate your project’s M&E plan and ensure that all necessary elements are included in the document.
- Think about relevant points of disaggregation for your project. Considering points of disaggregation when developing data collection tools will allow you to break down information into smaller, more specific subsets or categories when examining project results. In resettlement contexts, this often can include gender, location, language, race, legal status, or religion.
- Plan to gather and incorporate client feedback. See Switchboard’s guide on Collecting Client Feedback Data for a closer look on how to choose appropriate methods for collecting client feedback, and check out Switchboard’s webinar on Practical Strategies for Collecting and Incorporating Client Feedback Data to learn some two-way communication strategies, including client advisory councils and working groups.
4. Go Beyond Monitoring: Explore Project Evaluation
Depending on your project and its overall goals, properly measuring success may require conducting a project evaluation. Evaluation is the process of collecting and analyzing information once or twice during a project to assess a project or program’s outcomes and the factors that influence results. It involves separate steps from routine monitoring, which is the process of regularly and systematically collecting and analyzing information about a project and using it to adjust project implementation, enable internal and external reporting, and inform project design
Types of evaluation to explore include:
- Implementation or Process Evaluations, which answer questions like, “Were the activities of the project implemented as planned?” or, “What factors influenced the project results?”
- Outcome Evaluations, which answer questions like, “Do we see the changes that we were expecting to see? How significant are these changes?” or, “Were there any unintended results?”
- Impact Evaluations, which answer questions like, “Did the project produce better results than if there had been no project? Did it work better than other projects?” or, “Did we see the expected changes, and can we attribute those changes to the project?”
By asking these questions—and finding their answers in the data—project staff can gauge the effectiveness of programming from multiple complimentary perspectives.
Conclusion
By developing a strong theory of change, aligning outcomes to purposeful indicators, preparing for effective data collection, and planning ahead for evaluations, project staff can better position themselves to measure their projects’ success. Check the Switchboard website for more resources on monitoring and evaluation. You can also submit a technical assistance request to get assistance from Switchboard’s monitoring and evaluation team.
Dive Deeper!
Ready to learn more? Take Switchboard’s self-paced e-learning series on monitoring and evaluation fundamentals:
