In 2015, NRC undertook an evaluation of its shelter programme in Ethiopia. The evaluation was undertaken by NRC staff from the regional Horn of Africa office. The steering committee has reviewed the quality of evidence in the report and assesses it to be of high quality and based on robust evidence. There is no indication that the evaluation team has been biased or influenced in their findings, despite being an internal team.
NRC has been in Ethiopia since 2011, providing Shelter, Education, WASH and Food Security assistance to refugees in 14 camps located in 4 areas of the country. Shelter comprises the biggest component of the all assistance provided. The evaluation sought to examine all shelter projects implemented by NRC between 2011 and 2014. The initial purpose of the evaluation was to identify important areas of learning and document successes that can be replicated, as well as to highlight shortcomings to avoid in the shelter programme. However, findings throughout data collection shifted focus of the outcome towards key areas in need of improvement, quality improvements and assurance.
Following ALNAP’s adapted OECD/DAC evaluation criteria, the exercise sought to examine the different shelter solutions implemented in the different locations over the four years. This also included community infrastructure, such as transit and reception hangars that are used as communal shelters, and other technical infrastructure and services (mainly sanitation) provided by NRC for refugees (and host communities) in the four regions where NRC has been in operation. Evaluation data was gathered using a combination of household surveys (784 respondents), focus group discussions (19 no), key informant interviews (28 no.) and observations. The sampling of informants took into account variations in the different groups assisted (refugees and host community women, girls, boys and men), the different camps / locations and the varied types of shelter support provided.
The findings of the evaluation provide guidance for future program direction by drawing recommendations that will be used to improve overall programme quality and management.\
FINDINGS
Impact: The shelter programme has positively influenced the lives of beneficiaries in ways that stretch beyond the initial expected outcomes of the different individual projects. These range from the more basic and immediate physical protection against the elements and privacy, to improved access to food security and livelihoods, and savings in rent. Some beneficiaries identify with a reduction in vulnerabilities, affirming an improved sense of wellbeing and find themselves in a better position to cope with challenges of a refugee situation.
Relevance:
NRC has been operational in the 4 areas during critical influxes of refugees and helped address basic local needs that matched beneficiaries’ priorities. Most beneficiaries stated shelter to be a basic need that took precedence in their lives especially when they first arrived in the camps, confirming the self-evident need for shelters for newly arrived refugees. Only few (6%) could think of alternative assistance to shelter at the time delivered. With minor exceptions to Dolo Ado, target populations have largely received, and used shelters constructed by NRC as intended.
Effectiveness:
NRC is seen as a very responsive and effective organization that over the 4 years demonstrated capacity to deliver on its promises. It is also seen as proactive, and a leading provider of shelter solutions in part due to its being present in nearly all the camps while providing complementary assistance in a range of sectors. The organization has been praised for its ‘adaptability’, or more specifically, flexibility, in Gambella during the south Sudanese refugee influx of 2014.
Huge shelter gaps remain, at over 60% in Gambella. Current shelters are reported and observed to lack adequate space and privacy, and have a lifespan shorter than initially thought, while there is lack of maintenance support and follow up. Lately, NRC has come to be viewed as widely but thinly spread on ground to be effective, and now (2015) faces concerns of shelter quality.
Efficiency:
NRC is seen as a very efficient and responsive organization over the 4 years, outperforming most other agencies assisting refugees but needs to address the concerns of shelter quality by addressing 1) delays in materials procurement and challenges in handling, 2) shortcomings in shelter design and costing, 3) limited supervision and quality control, and 4) management of fleet for transportation of staff and materials.
Coordination:
Coordination with UNHCR, ARRA and other humanitarian agencies as well as within NRC is reported and observed to work well, to the extent that there is no reported duplication of assistance. In most places, NRC has often been the sole shelter partner of ARRA & UNHCR, limiting coordination challenges. It is however affected by the limited presence of supervisory technical and middle management staff in the camps / sites.
Cross cutting issues:
the organization is evidently aware of the associated environmental, and protection (age, gender and disability) concerns and has clearly mainstreamed them into the shelter programme. However, a few shelter related protection concerns facing unaccompanied minors (UAMs) in Shire are highlighted for further action.
RECOMMENDATIONS
To increase NRC’s technical and managerial presence on ground, needed to address the reported and observed shelter quality concerns, the following actions are recommended;
1. Seek ways of working with more refugee incentive workers. Consideration and ways of increasing community participation beyond the current form and levels is also urgently needed. Towards this end, NRC needs to see how to address with ARRA and UNHCR the concern of having shelter beneficiaries selected and allocated shelters /plots before start of construction. This may require negotiating with UNHCR and ARRA to be tasked with beneficiary shelter allocations. Equally important, NRC needs to advance and sell better the idea of owner driven shelter construction to the refugees, ARRA and UNHCR.
2. Undertake a review of the designs and costing of all the shelters constructed for refugees in all the four areas. The shelter designs are to be reviewed, to allow for expansions and use of alternative materials that can better withstand termite attacks for a more elongated lifespan. Further, a joint review of the shelter design, by child protection and shelter experts which should benefit from the input of the minors is urged, to address the protection concerns raised by the minors.
3. Develop a closer and more practical way of having the shelter programme staff work with the support functions, especially on effective material procurement and handling, and fleet management. This includes consideration for long term agreements for materials supply, and appropriate warehouses for early procurement and stocking of commonly used shelter materials. A more targeted training on budget monitoring and management should be developed and held for key programme staff tasked with implementation and monitoring of budgets.
4. Strengthen the current M&E system and practice with a particular focus on quality management.
5. NRC needs to review its wide but thin presence in the various camps it’s currently operational, especially Gambella, as well as the portfolio of activities, to be in line with the available funding, or resources, and the capacity to deliver quality programmes.
LESSONS LEARNT / CASE STUDIES
The evaluation identified two case studies with implications beyond the country programme. These include 1) Use of YEP to address shelter skill gaps in Shire, 2013 and 2) NRC’s adaptability in Gambella, during the South Sudanese refugee influx of early 2014.
Evaluation