Who do we help in Viet Nam and how? A seemingly simple UN evaluation question…

As a child, my mother took me to the doctor because I was limping. There she learned, to her relief, that my only problem was my shoes were too small.

As I reflect on the recent UN experience evaluating Viet Nam’s One Plan 2012-2016 this story comes to my mind. Maybe our evaluation shoes are just too small?  From the very beginning of our efforts to design an evaluation of the UN’s contribution to development in Viet Nam in that time frame, it was clear we needed a different approach. But which one?

The first task was to assess the information regarding the UN work in the last four years. We confirmed our worst fears: so far, only a small amount of evidence had been generated by UN agencies.

So there we were, with limited resources, asking ourselves: How can we evaluate the last four years of the UN work in Viet Nam in a feasible way? Or, as we evaluators say, how can we find a “common thread”?

Trying on a new pair of shoes: who did we help?

Vulnerable and disadvantaged populations were the focus between 2012-2016.  While no evaluation is perfect, our approach was to focus on a number of illustrative case studies addressing the rights of the most disadvantaged populations hoping that this would allow us to measure our impact in an innovative way.

To what extent had or had not the UN delivered results for particular parts of the Vietnamese population through policy advice? To what extent were we brokers of ‘knowledge exchange’? Did the UN conveyed stakeholders bringing international expertise to the table?

We quickly got the management buy-in for the overall idea but we were to learn that this was the beginning of our challenges.

Roadblock #1: Which case studies will you evaluate?

The independent evaluation team hired to conduct this exercise took on the difficult task of identifying potential case studies. As a starting point they identified the following vulnerable populations groups: ethnic minorities, rural migrants in urban settings, people living with HIV and populations vulnerable to climate change and natural disasters.

However, once shared with UN colleagues, they suggested that a much deeper analysis of vulnerabilities was needed. In a way we were trying to define a concept of who vulnerable populations are, and this is a complex, interconnected dynamic which doesn’t necessarily fit into boxes that are easy to evaluate.

Roadblock #2: Case studies won’t reflect the work of different UN agencies in Viet Nam

The evaluation team’s initial selection of case studies proved to be not only limited in scope but also not representative of the work of the different UN Agencies in Viet Nam. With this in mind, it was decided to expand the cases and reflect the work of as many UN Agencies as possible.

Eventually we choose four themes for the case studies:

1. Support to ethnic minorities
2. Opening space for citizen participation
3. Evidence-based advocacy
4. Capacity building to promote resilience and reduce risk 

Roadblock #3: Don't bite off more than you can chew

In total, UN Agencies proposed 30 initiatives (most of them joint initiatives) to inform the case studies. We were more inclusive, but expanding the scope meant reducing the depth of the analysis.

What did we learn using these new pair of shoes?

Well… we were not limping anymore, but we did get some blisters from our new shoes.

Here are some lessons learned from this experience:

1. As we tried to identify who in Viet Nam we helped, we realized that we needed a more sophisticated analysis of the vulnerability concept in Viet Nam, a gap we already addressed in our new common country assessment.
2. The importance of considering “emerging vulnerabilities” such as those experienced by the elderly, the disabled or the ‘missing middle’ - those in the middle of the income distribution whose incomes are too low or insecure to be resilient to shocks.
3. We need to identify new “macro layers of vulnerabilities” like environmental and urban vulnerabilities.
4. We need a coherent system-wide strategy to address the needs of a given group. For instance it is clear that the UN is doing a lot in support of ethnic minorities but not necessarily in a coordinated or holistic manner.
5.  A less ambitious scope for the case studies could have shown the real connection between the upstream policy work of the UN and the benefits for specific vulnerable groups.
6. Last but not least, more interviews can help triangulate self-reported results making more visible the ‘human face’ of the UN work in Viet Nam.

Maybe we need a pair of flip-flops

As the 2030-Agenda promise of ‘leaving no one behind’ is being rolled-out and localized across the globe, we need to regularly assess the UN’s contribution to it. For this, credible and timely evaluations which unpack vulnerability and demonstrate contribution towards real changes in people’s lives will be critical.

We need to keep looking into innovative approaches to evaluate the UN work. In short, we need new shoes and they have to be as colorful, flexible and down to earth as a pair of flip flops. Anyone out there has been window shopping? Let us know!