Development practitioners – donors, designers, policy shapers and implementers – across the globe agree that evaluation must play a stronger role in shaping development efforts. For some time now, we have heard calls for “evidence-based” policy-making; for the professionalisation of evaluation practice as well as for the need to increase the monitoring and evaluation (M&E) capacity in developing countries. Indeed, after more than 50 years of global development efforts, and two years after our first global deadline (the Millennium Development Goals), we’re still far away from meeting the targets we set ourselves to achieve in 2015.

MDGs

How should evaluation in practice be different to be able to respond to the challenge? Let’s start by reflecting on the most recent developments already shifting the way evaluation is conceptualised and applied in practice:

The sustainable Development Goals: a new playing field for development

Evaluation approaches to the SDGs … must have a broader, holistic, and more complex approach than for the MDGs.

The advent of the Sustainable Development Goals (SDGs) adopted in September 2015 meant the beginning of a new era for global development efforts. While the SDGs broadly incorporate themes present in the MDGs, “they also reflect the growing concerns about the future of our planet as reflected in the focus on sustainability and the need for social inclusion to ensure No one is left behind[1]. The OECD acknowledges that “the universal nature of the Sustainable Development Goals (SDGs) calls for a profound transition in the way we look at – and work for – development: from a focus, mainly on the needs of poor countries, to one that emphasises well-being and sustainability in all countries”[2].

Evaluation approaches to the SDGs therefore, must have a broader, holistic, and more complex approach than for the MDGs. SDGs bring interconnectedness and complexity: we have moved from 8 MDGs to 17 SDGs and from 48 to 165 measurement indicators. In addition, the fact that the SDGs are based on five universal and interlinked values—people, planet, prosperity, peace and partnership—means that the achievement of individual SDGs cannot be assessed without considering their linkages and dependencies to other SDGs. The SDGs recognise that most development programmes are complex and require a multi-prong approach in their design, implementation and evaluation. We know, for example, that increased complexity makes the measurement of causal relationships very difficult and, in some cases, impossible. Contribution analysis and variations such as outcomes harvesting and outcomes mapping, recognise that in complex programme contexts, it is not possible to use attribution analysis, and the best we can do is to assess the contribution of different interventions to the observed changes[3].

‘Old-new’ cross-cutting issues: equity and gender

The SDGs are strongly anchored in a social justice approach. In practical terms this means that the objective is for member-states to work towards extending the reach of services to those populations traditionally excluded or marginalised, including and especially women and girls, ethnic and religious minorities, refugees, illegal immigrants and people differently-abled. From an evaluation perspective, the emphasis on equity means  going beyond aggregate indicators, to proactively uncover situations and/or population groups with no access to services. In turn “a focus on the identification of groups who have been excluded will require governments to acknowledge that their social and economic performance has not been as strong or effective”. This is what lies behind the spirit of ‘no one left behind’.

From an evaluation perspective, the emphasis on equity means  going beyond aggregate indicators, to proactively uncover situations and/or population groups with no access to services

One of the implications of the renewed gender focus is the need to understand how programme implementation is influenced by social, cultural, political, physical and economic factors that are embedded in specific contexts. From an evaluation perspective, a gender focus will require that the gender lens is applied at all stages of the evaluation process, i.e. the household might not be appropriate as a unit of analysis for data collection and analysis given that as we know, not all household members have equal access to resources.

New assessment criteria for the 21st century

Several voices are calling for the re-evaluation of the use of DAC criteria[4] for evaluation. In a recent blog, the Independent Development Group argued that “effectiveness seems rather lenient”, given that objectives against which programmes are measured, are often set by programme managers in the first place.  The reality of increased interconnectedness and complexity “might change the nature in which objectives are set, which will either make it more challenging to assess whether they were met, or demand an equally dynamic evaluation tool, or both”. Similarly, Imani Development Managing Director, Andrew Simpson, has recently argued that DAC criteria are insufficient for the monitoring and evaluation of social innovation programmes, because they fail to value the crucial relationship between risk and impact in the development area.

Evidencing Value for Money

Overseas development assistance and FDI inflows in many developing countries have been dwindling in recent years. Decreasing financial flows for development will call for improved accountability by foreign governments pressed to justify financing for development to their tax payers. More emphasis will be placed on impact and value for money evaluations to justify any continued support. This renewed emphasis will require more trained practitioners and possibly, adjusted tools, which are responsive to the new demands. ActionAid’s new Value for Money guide is such an example.

Data revolution

In the last 10 years, we have witnessed a great expansion in the use of digital technology in the collection, analysis and reporting of information, from the wide-spread use of smart phones and applications for data collection and GIS mapping to the new data visualisation techniques used for reporting and dissemination. According to the technology and innovation giant IBM, a staggering 90 percent of the data that currently exists today was created in the last two years[5]. One of the most powerful benefits of big data is the potential for aggregate datasets to become increasingly available to development organisations; this is driven by the promotion of greater transparency, sharing and exchange of data across international and local organisations. The Humanitarian Data Exchange or the Earth Observation for Development project are such examples.

To realize the global development potential of big data, connectivity and internet penetration in low-income countries must substantially improve. Now, 2/3 of people living in developing countries are offline (4 billion people); in the countries where does connectivity exist, average monthly fixed-broadband prices (in PPP$) are 3 times higher than in developed countries and mobile-broadband prices are twice as expensive as in developed countries.

From a programming perspective, bridging the digital divide between high-income and low-income countries will be the priority; from a research and evaluation perspective, prioritising collaborations around the accessing and sharing of data is a must in the complex and interconnected environment we live in.

Conclusion

Evaluation experts[6] across the globe agree that the complexity of the world’s development challenges call for a formidable expansion of monitoring and evaluation systems. This will involve a strengthening and rethink of evaluation efforts to make use of new tools and opportunities, and to adopt old and new evaluation criteria with renewed impetus and purpose.

In answer to the primary challenges of evidence and capacity posed earlier, Imani development is well placed to contribute.  Our adaptation of the Sustainable Livelihoods Framework as an evaluation tool has generated a dynamic tool that is responsive to the complexity and interconnectivity of the SDG’s.  Our decades of experience in regional integration and more recent focus on the role of women in trade, places us on the front of the wave as we navigate cultural and community sensitivities yet driving an inclusive and dynamic response to trade constraints.  The issue of data volume, reliability and appropriateness is a challenge of infrastructure, design and human capacity.  Lastly, our Africa-wide network of local consultants are a testament to our commitment to indigenise local capacity in the continent.

While feel confident in the part we are playing many of the areas we operate in continue to be underserved and resultant fragilities unresolvable – investment and innovation will lead the way… we will be part of it.

About Elena

Elena_Mancebo_Masa

Elena Mancebo is a Senior Consultant at Imani Development, where she focuses on planning, monitoring, evaluation and learning. Over the past nine years she has led evaluations for governments, non-profit and development aid organisations working primarily in the African continent. Some of her present and past clients include the Western Cape provincial government in South Africa; Comic Relief; Development Bank of Southern Africa; or USAID. As a skilled evaluation designer and implementer, nothing has served her so well as her empathy, adaptation capacity and relationship-building skills.

She holds a BA (Honours) of Social Science in Political Science from Madrid, where she is originally from. The future of African cities is her latest passion, and she’s currently pursuing a PhD in Sustainable Development Planning at Stellenbosch University the Western Cape.

References

[1]  Michael Bamberger, Marco Segone and Florencia Tateossian: Evaluating the Sustainable Development Goals. With a “No one left behind” lens through equity-focused and gender-responsive evaluations. 2016

[2] OECD: The Sustainable Development Goals: An overview of relevant OECD analysis, tools and approaches.

[3] Michael Bamberger, Marco Segone and Florencia Tateossian: Evaluating the Sustainable Development Goals. With a “No one left behind” lens through equity-focused and gender-responsive evaluations. 2016. Page 48

[4] Criteria set out by OECD countries in 1991 to evaluate the performance of development programmes. The criteria are relevance, effectiveness, efficiency, sustainability and impact.

[5] Solving the data conundrum: How to leverage tech and ‘big data’ for impact

[6] See Zenda Ofir’s blog on  and Rockefeller Foundation, Evaluation Office on 5th wave of Social Impact Evaluation

Michael Bamberger, Marco Segone and Florencia Tateossian: Evaluating the Sustainable Development Goals. With a “No one left behind” lens through equity-focused and gender-responsive evaluations. 2016

https://www.evalpartners.org/sites/default/files/documents/evalgender/Eval-SDGs-WEB.pdf

OECD: The Sustainable Development Goals: An overview of relevant OECD analysis, tools and approaches

WBG: How can development partners meet the huge demand for evaluation capacity?

IWI: From MDGs to SDGs- The Journey Towards Gender Equality

ITU: Facts and Figures 2015

IPS: G77 Expresses “Deep Concern” over ODA and FDI Declines in Africa