Measuring impact may be challenging in the complex landscape of development initiatives, but it is fundamental to ensuring efficient resource allocation to achieve sustainable impacts.
Researchers often rely on experimental designs and drawing inferences primarily on quantitative data, as the standard in evaluating development interventions. These are the conventional impact assessment approaches. While these are valuable, they often fail to capture the complexity of social change processes. Conventional approaches tend to focus on measuring outcomes at a specific time, overlooking the intricate pathways and mechanisms that lead to potential outcomes. Traditional impact assessments do not always account for contextual influence and other various factors shaping change.
Process tracing as a learning journey
Quantifying results is crucial, however, understanding how and why observed outcomes occur is equally essential for holistic impact assessment and innovation scaling. Process tracing plays a significant role in tracking causal mechanisms from the initial causes to the ultimate effects. It maps out key events and actors while collecting specific evidence for each mechanism hypothesized to have resulted in changes.
At the outcome level, changes are observed in the agency and actions of system actors such as government ministries, agri-food businesses and farmers, who have been informed or influenced by research outputs including trainings, workshops and reports with novel scientific findings. Process tracing, therefore, requires evidence examination in a given case study to systematically identify and test the sequence of events, activities and decisions contributing to observed results.
Process tracing versus conventional methods
Conventional impact assessment methods measure net effects by comparing outcomes between treatment and control groups. These include randomized controlled trials and quasi-experimental designs and rely on quantitative data to establish causal relationships through counterfactual analysis of a mental simulation where you think about something that happened, and then imagine an alternate ending. They often assume linear causality and fixed intervention pathways, which may fail to account for complex social change dynamics.
Process tracing adds a nuanced understanding of causality by moving beyond correlations and experimental counterfactuals to identify the specific mechanisms driving changes. This method is particularly well-suited to evaluating interventions in dynamic contexts, where multiple factors and actors interact to shape outcomes.
Process tracing enables us to identify unintended consequences and unexpected pathways of change. This provides valuable insights for adaptive management. Its emphasis on qualitative data and in-depth case studies allows for a richer and more contextually grounded understanding of impact.
Though largely effective, process tracing can, however, be time-consuming and resource-intensive, requiring extensive data collection and analysis. The quality of a process tracing study, meanwhile, relies heavily on the availability, proof value and quality of data, which can be challenging to obtain.
Good data collection requires good relationships and commitment based on a clear understanding of the method. It also needs trust and dedication from the evaluators and team members. Interpretation and identifying causal mechanisms can also be subjective, requiring careful consideration of alternative explanations and accounting for potential biases. A standardized approach is needed to overcome challenges related to data accessibility and subjective bias among researchers.
Promising innovations going forward
Process tracing offers a promising method for enhancing process-based development initiatives. Although resource-intensive, the approach enables researchers to identify and leverage emergent change pathways and refine strategies based on in-depth nuanced insights.
By embracing process tracing, projects, including those aiming to address the world’s water challenges, can more effectively trace intricate processes of social change, ultimately leading to more impactful innovation scaling in complex and dynamic environments. Although process tracing is not a replacement for experimental impact evaluation, it does have the potential to generate precious insights into innovation scaling processes and the research methods beyond, behind and beside experimental studies.